phplist bounce processing issue

Based on a complaint received on 19 Nov, checked the server logs and found that Gmail is starting to mark our mails as spam because we're daily sending mail to bouncing gmail accounts. 

PB went to the reconcile users page in phplist and looked at the process of unsubscribing users who have 15 and more bounces. (Found about 9,527 users) - but did not run it since this was not 15 consecutive bounces. Running the bounce processing manually, the bounce processing takes 3 hours, identifies the bounces, but does not unconfirm subscribers with 5 consecutive bounces. So, running bounce processing is not solving the problem. We needed to fix the bounce processing rules which are being applied.

PB found, checking for the location where the rules are defined and found that phplist is picking up rules from the database table. Table names: our_prefix_bounceregex and our_prefix_bounceregex_bounce. But those tables showed empty records. Did not find any web interface to add rules to these tables. Did not find a backup with entries. PB downloaded the sourcecode (version 2.10.10) from https://sourceforge.net/projects/phplist/files/phplist/ and checked the phplist.sql file that creates the database on a new installation. There too the records were empty.

(Edit: Found a discussion about this in the phplist manual here. )

Then, a quick and dirty fix implemented by PB on my request: in phplist web interface "view bounces" selected the message id 4189, got the list of email ids that had bounced for the message id 4189

mysql> select id,email,confirmed from our_prefix_user_user where id IN ( select user from our_prefix_user_message_bounce where message=4189);

- about 1147 records

Unconfirmed the users with the following sql update:

mysql> UPDATE our_prefix_user_user SET confirmed=0  WHERE id IN (SELECT user FROM our_prefix_user_message_bounce WHERE message=4189);

Query OK, 1146 rows affected (0.54 sec)

Rows matched: 1147  Changed: 1146  Warnings: 0

compiling siril

Following the dependencies listed at
https://free-astro.org/index.php/Siril:install#Dependencies
I did the following to install dependencies on Linux Mint 18.1 -

sudo apt-get install gtk3*
sudo apt-get install cfitsio*
sudo apt-get install fftw3*
sudo apt-get install gsl*
sudo apt-get install libconfig++*
sudo apt-get install libraw*
sudo apt-get install libffms2*
sudo apt-get install libtiff*
sudo apt-get install libjpeg*
sudo apt-get install libpng*

Then running autogen.sh complained -
./autogen.sh: intltoolize: not found


and for that I did
sudo apt-get install intltool
sudo apt-get install libgsl*

This was enough to build it using
./autogen.sh
 make
 sudo make install

Thought about doing a full-fledged fork and pull request as in
https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/

For that, I needed to set the ssh key, as in this stackoverflow post,

cat ~/.ssh/id_rsa.pub
and copied the key to github -> settings -> ssh key

git clone git@github.com:my-account-name/Siril-0.9.git
cd Si*
git remote add upstream git@github.com:lock042/Siril-0.9.git
git checkout master
git pull upstream master && git push origin master
git checkout -b feature/stackallmedian

It turned out that this version had a bug where stackall would cause a segmentation fault. This was fixed in version v0.9.8-beta3 which also had a stackall median command, which did not do normalization by default, so my problem was solved and I did not have code it myself to do the git pull requests and so on.

more on SER, FITS and Octave

Adding to my earlier post - Siril had a stackall command which stacks all the SER files in a folder to sum-stacked FITS files. That would have taken away the drudgery of having to open 830 SER files manually one by one and converting them though the creation of directories could be automated by

mkdir $(seq --format 's%03.0f' 1 830)

(from
https://unix.stackexchange.com/questions/48750/creating-numerous-directories-using-mkdir )

which would create
s001
s002
etc upto
s830.

Unfortunately, Siril's stackall does sum stacking, and (at least according to my understanding, in version 0.9.7) was integer overflowing or showing some such behaviour - though the developers mentioned that it is supposed to convert to 64 bit, and then normalized to 16 bit. Maybe the normalizing was what was causing the issue. My data set had numbers like 44,000 per pixel and 102 frames, resulting in sums like 4,400,000 - even signed 32 bit int would only overflow at 2,147,483,647. Anyway, using the stackall sum stacking, I was getting plots like this

instead of plots like this.

Possible solution may be to implement a median stacking function for siril, like stackallmedian.

Edit: The siril team has now implemented median stacking using the command stackall median - and current branch has no normalization applied, so it works well for me :)

compiling QHYCCD camera SDK sample application with OpenCV

Steps needed to cleanly compile the LiveFrameSample code from QHY camera SDK, once OpenCV is installed -

Tested the opencv installation, compiling and running webcam capture test code from http://henryhermawan.blogspot.in/2007/09/capturing-frames-from-usb-camera-using.html

Followed the INSTALL file and compiled + installed the QHYCCD SDK.

cmake -DCMAKE_INSTALL_PREFIX=/usr .

make

sudo make install

Tried the sample without modifications -
cd sample
mkdir build
cd build
cmake ..
make

LiveFrameSampletest, SingleFrameSampletest and ControlCFWtest compiled without errors, but for CaptureDarkFrametest errors were reported, like
CaptureDarkFrame.cpp:76:83: error: invalid conversion from ‘int*’ to ‘uint32_t* {aka unsigned int*}’ [-fpermissive]
         ret = GetQHYCCDChipInfo(camhandle,&chipw,&chiph,&w,&h,&pixelw,&pixelh,&bpp);
and so on.

Anyway, I was only interested in the frame capture tests, so I created LiveFrameSampleOCV.cpp from LiveFrameSample.cpp and made the following changes -

1. Uncommented the #define for opencv support and made that line
   #define OPENCV_SUPPORT 1
2. Changed the set resolution line to
   ret = SetQHYCCDResolution(camhandle,320,240, 320, 240);
   (first two arguments after camhandle are dimensions, next two are position)
3. Defined an int key before the display loop, and changed the display loop to an infinite loop -
   //ret = QHYCCD_ERROR;
   //while(ret != QHYCCD_SUCCESS)
   while(1)
4. Added a break on escape key,
   key=cvWaitKey(30);
   if (key == 27)
   break;
5. Modified the CMakeLists.txt to uncomment
   find_package(OpenCV REQUIRED) and
   include_directories($(OpenCV_INCLUDE_DIR))
   and also changed the link libraries line to
   target_link_libraries(${FILENAME1} -lqhy ${LIBUSB_1_LIBRARIES} ${OpenCV_LIBS}
   where the OpenCV_LIBS have been added. 
6. Ran cmake .. from the build directory again, and then
   make LiveFrameSampleOCVtest

So now ./LiveFrameSampleOCVtest shows a live display of the camera at 320x240 resolution, around 20-24 fps. 

While running cmake, it shows
 -- Found libusb-1.0:
--  - Includes: /usr/include/libusb-1.0
--  - Libraries: /usr/lib/x86_64-linux-gnu/libusb-1.0.so

But could not build LiveFrameSample without using cmake - the gcc command needs some more work - this does not work -

gcc -Wall LiveFrameSampleOCV.cpp -o LiveFrameSampleOCV -l/usr/lib/x86_64-linux-gnu/ `pkg-config --cflags --libs opencv usb-1.0`

installing the latest OpenCV with Python bindings on Linux Mint 18.1

Followed the Ubuntu 16.04 tutorial by Adrian Rosebrock with some changes. Since my installation had python using anaconda, I used conda install instead of pip install. And since my first attempt at creating a virtual environment failed, I went ahead with the installation without a virtual environment. Most of the requirements were already installed. My cmake string also had a different path to the python executable due to anaconda.

sudo apt-get update 
sudo apt-get upgrade
sudo apt-get install build-essential cmake pkg-config
sudo apt-get install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev 
sudo apt-get install libgtk-3-dev
sudo apt-get install libatlas-base-dev gfortran
sudo apt-get install python2.7-dev python3.5-dev

OpenCV stable version is now 3.3.1, so

wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.3.1.zip

unzip opencv.zip

wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.3.1.zip

unzip opencv_contrib.zip

conda install numpy

cd opencv-3.1.0/

mkdir build

cd build

cmake -D CMAKE_BUILD_TYPE=RELEASE \

    -D CMAKE_INSTALL_PREFIX=/usr/local \

    -D INSTALL_PYTHON_EXAMPLES=ON \

    -D INSTALL_C_EXAMPLES=ON \

    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.3.1/modules \

    -D PYTHON_EXECUTABLE=/home/mac/anaconda2/bin/python \

    -D BUILD_EXAMPLES=ON ..

cmake rapidly went through its checks, then I had to do the build,

make

which took around half an hour judging by the timestamps - I was doing other things.... and then finally 

sudo make install

sudo ldconfig

Verified by doing

ls -l /usr/local/lib/python2.7/site-packages/

which showed up cv2.so

and finally linked this cv2.so to the site-packages folder, in my case

cd ~/anaconda2/lib/python2.7/site-packages

ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so

Tested by opening a python shell, importing cv2 and checking its version,

python

 >>> import cv2
>>> cv2.__version__
'3.3.1'
>>>

Edit: August 2019 - updated instructions for installing on Ubuntu 18.04 - 
https://www.pyimagesearch.com/2018/05/28/ubuntu-18-04-how-to-install-opencv/

 

SER, FITS and Octave

While capturing a series of images in an autocorrelation experiment using a QHY5L-II and Sharpcap, instead of capturing as a series of PNG files, captured as SER files by mistake. From the experience of trying to use that data, found the following.

1. SER capture is faster than PNG sequence capture - PNG was capturing at 20 fps or so, but dropping frames after 80 frames or so, and was taking the last 20 frames at around 5 fps. SER was not dropping frames at all. All this on the Menlo systems Lenovo laptop. 
2. Octave and Matlab don't open SER files without conversion, the definitive site for SER files seems to be http://www.grischa-hahn.homepage.t-online.de/astro/ser/
3. After some back and forth on this thread, found that Siril can export SER to a sequence of FITS images without losing the 16-bit resolution. 
4. Imagemagick can be used to convert the FITS images to 16-bit TIFF using
   mogrify -format tif *.fits
5. Alternately, the fits octave package can be used to read the data into Octave from the FITS files. Although Octave uses Imagemagick to implement imread, imread doesn't seem to work with the FITS files from Siril, complaining
   error: Magick++ exception: Magick: Unexpected end-of-file (/media/mac/SAM2TB/Autocorr/2017-10-16/Capture/14_10_49/Capture_00002.fit) reported by coders/fits.c:370 (ReadFITSImage)
6. In order to install the fits package, had to install its dependency, with
   sudo apt-get install libcfitsio*
   and then run octave as root,
   sudo octave
   and within octave, run
   pkg install -forge fits
7. Reading using the fits package and imread give different results.
   imread returns uint16 rows x columns
   read_fits_image returns double columns x rows, upside down.
   So, for equivalence with imread, we need to use flipud.
   b=read_fits_image('im.fits')
   b=b';
   b=flipud(b);
   imagesc(b)
   gives the same numbers as
   a=imread('im.tif')
8. identify Capture_0035.png
   can be used to check whether the png is 8-bit or 16-bit. 
9. Installing ISIS and SER Player turned out to be dead-ends, since they did not convert SER to PNG without converting to 8-bit.

Edit - more on siril's stackall here. 

removing malware/adware from Intex phone - trois

Even after my earlier posts about removing malware from Intex phone, one issue persisted. Occasionally, Rs. 1.50 would be deducted as SMS charges. At first I thought it was some sort of location finding behaviour by some app - as I noted, this deduction was noticed after I installed Uber and PayTM apps. But even after uninstalling those apps, the issue remained - deduction of Rs. 1.50 once in a week or so.

Finally I contacted my carrier Airtel via email - which seems to be the only way to reach their customer service department. They advised that I could Dial *121*51# (Toll Free) or 12181 (Toll Free) to check last 5 deductions which have happened in the last 72 hours. Using this information, I found that the SMS charges were to a number 9582943043. Googling, found that this is an "activation feature" on Intex phones!

https://together.jolla.com/question/139190/intex-aqua-fish-re-activation-sms/

Now I seem to have removed the offending Intex services - using Kingroot and root uninstaller - searched for sms in system apps, found SmsRegister - com.android.smsregister - uninstalled it. This seems to have solved the issue - no sms sent for the last one month. 

a tale of redirects

I added a second domain to Godaddy's deluxe hosting, using My Products -> Web Hosting -> Manage All ->  Manage (under primary domain name)  -> More -> Hosted Domains. The root folder for this was configured like /seconddomain, but when trying to access the second domain, was getting a redirect to
http://firstdomain.org/seconddomain

Godaddy support, over the course of 3 days, told me that this was due to some redirects in the .htaccess file. But they could not fix it for me. Finally when I tried editing the .htaccess file, found that the issues were due to some redirects meant for the CodeIgniter framework, redirecting html files to their php equivalents. The original .htaccess was
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?$1 [L]

In order to make it work with the second domain, I added some conditions, and made it

# BEGIN WordPress

RewriteBase /

# END WordPress

RewriteCond %{HTTP_HOST} ^www.seconddomain.org$ [NC]
RewriteRule ^(.*)$ http://seconddomain.org/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^seconddomain.org$ [NC]
rewritecond %{REQUEST_FILENAME} !-f
rewritecond %{REQUEST_FILENAME} !-d
rewriterule ^(.*)$ /index.html [L]

RewriteCond %{HTTP_HOST} ^www.firstdomain.org$ [NC]
RewriteRule ^(.*)$ http://firstdomain.org/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^firstdomain.org$ [NC]
rewritecond %{REQUEST_FILENAME} !-f
rewritecond %{REQUEST_FILENAME} !-d
rewriterule ^(.*)$ /index.php?$1 [L]

Now both sites firstdomain.org and seconddomain.org seem to be working fine. www.firstdomain.org redirects to firstdomain.org and similarly www.seconddomain.org redirects to seconddomain.org.

removing duplicates

Found some duplicate filename entries in our schedule page remote database. PB found that these were due to some files with filenames beginning with a carriage return - ASCII 13 -

and he found there were 144 such duplicated files.

I found that these had been added in the early days - double digit index numbers - and so could be deleted - did

SELECT * FROM `name_of_table` where LEFT (fileName, 1) = '\r'

and then deleted all those 141 records using the phpmyadmin interface.

fulldome tools links

A, an alumnus, sent me some links -

http://voicesofvr.com/67-jane-crayton-on-using-vdome-to-create-immersive-dome-and-live-vj-experiences/ 

http://zubr.co/work/dome/

http://www.fddb.org/inside-dome-new-plugin-for-touchdesigner/

http://omnido.me

Intex Aqua 4G Strong and Airtel 4G - solved

Unfortunately, my Intex Aqua 4G Strong handset does not seem to work with Airtel 4G network. The issue could be that Airtel's 4G is on LTE band 40 (2300 MHz) which does not seem to be supported by this handset. So, currently, even when Airtel4G is seen as an available network using Settings -> More -> Cellular Networks -> Network operators, and when chosen, shows "Registered on network", the status shows 3G, and apps like Coverage and Phone tester show the connected cell to be WCDMA / HSPA and not LTE.

Phone tester was earlier indicating that this handset was not reporting GPS connections. Maybe it lacks antennae for GPS and LTE band 40? Using Phone tester with the Mi3 would take a couple of minutes for first connecting to a GPS satellite, and later, after closing and opening the app, would connect within seconds even indoors.

Edit: I emailed 121 at airtel dot com with my 4G connection issue. They called back in the evening, saying that it was an issue at their end, "we would like to inform you that due to some technical issue, internet service on your mobile was not working. However, we have reset the internet service for your mobile.
We request you to restart the handset before using the internet service."

I restarted after seeing their email, and the phone connected to 4G  :)

updated budget airline flying experience

I had posted in 2007 about flying Air Deccan. Since Air Deccan no longer exists, maybe I should write about my recent trip on Spicejet.

Ticket price was supposed to be Rs. 1600 or so for CCJ to BLR, hand-luggage-only fare. Somewhere in the online booking process, I was charged some more, so it became Rs. 1900. Somewhat comparable to the Rs. 1800 (Tatkal) II AC railway ticket price.

Ticket and ID is required while entering the airport. Security check was relatively hassle-free, no lines when I arrived one hour before my flight time. No tag needed for hand-luggage. CISF personnel insist on paper boarding pass so that they can rubber-stamp it - whether web check-in or normal check-in - so the mobile boarding pass is not of much use. They also asked me to take out electronic items from my bag and put them on a tray for X-ray. The waiting lounge was full of passengers - Air India flight to Mumbai was four hours late, those passengers were being provided with breakfast in the lounge when we were boarding. Low-floor buses to and from the aircraft at both airports, a Cobus 3000 at BLR.

As noted in my previous post, the flight attendants are a bit rushed for time, trying to manage the very short turnaround times. They were doing the safety drill when the aircraft was rolling towards the start of the runway, just completing when it started the takeoff run. The Bombardier Q400 managed the 350 km trip in just 35 to 40 minutes of flying time. Another five minutes for taxiing and so on. A bit of turbulence over the Western Ghats as usual - maybe for five minutes or so. Also a big downdraft just before landing. Otherwise quite smooth.  The pleasant-mannered cabin crew managed to serve pre-booked meals and free water for quite a few passengers during this short flight.

The flight seemed to be full - I had chosen the last row window seat, 20D, for the little bit extra legroom. At nearly six feet, I was reasonably comfortable, with an inch or two to spare between my knees and the seat ahead. But the seat reclines only a couple of inches.

Overall a good experience, with only the taxi fare from the airport being an extra cost as compared to the train journey. 

upgrading from Korg Nanokontrol to Nanokontrol2

I mapped the MIDI CC signals of the Nanokontrol2 to Reaper following the method given at
http://hnsws.blogspot.in/2010/08/korg-nanokontrol-and-reaper.html

with the important difference that the Nanokontrol2 had to be plugged in keeping the
Cycle and Set 
buttons pressed to make it go into the MIDI CC mode, as mentioned in the instruction booklet supplied along with the device.

I tried rebooting the computer, the setting seems to be remembered. So, probably for daily use, this pressing of buttons at startup may not be needed.

In case you have issues with it, you may need to disconnect the Nanokontrol2, and connect it back keeping Cycle and Set buttons pressed.

(Specific to our use - other functions are similar as before, except that there are 3 buttons for each row instead of 2 for the earlier Nanokontrol - the lower buttons are mapped to SOLO for the slider tracks, the upper buttons are mapped to SOLO for the knob tracks. The middle buttons are left unused for now.)

splitting a multi-track wav file into separate channels in Reaper 3.5

Right-click on the track, choose Item processing -> Explode multichannel audio or midi items to new mono items 

Sunday, August 13, 2017

avoiding dead time on radio during live broadcasts

From an email sent to my colleagues.

It has been our practice to avoid dead time on the radio during mandir live broadcasts by playing the file BI.mp3 which can be found on the desktop of the mandir mac.

The procedure is -

• drag and drop the BI.mp3 into the track corresponding is knob 4 (input 12) ahead of time, if you anticipate a period of silence

• click on the record enable button - red button next to the fader on the onscreen mixer - to disable record enable (so that the mp3 gets played out instead of the input from that channel)

• fade in the music within 2-3 seconds of silence, and fade out similarly. 

• Once bhajans start (or some program starts where input 12 is needed) again click record enable to enable that input.
  
• Another point to be noted is: Please DO NOT save the project when you quit. (If recorded, the wav file is saved automatically.)

Wednesday, May 17, 2017

converting a FAT32 volume to NTFS and backups

With WCry and other malware in the wild, I wanted to take a full system backup of the Acer laptop. Decided to go with Windows built in backup - Start -> Run -> ntbackup

Around half an hour later, ntbackup complained that the drive it was writing to was FAT32, and does not allow files larger than 2 GB (or 4GB? I forgot). Decided to convert it to NTFS, retaining the data on it.

Ran convert from command prompt. It failed, complaining that there was insufficient memory. Moved over the existing files to another hard disk and then tried convert again. It again complained of insufficient memory. Probably a 2 TB volume is too much for it on a 32 bit system. So, Control Panel -> Administrative Tools -> Computer Management -> Disk Management. Deleted the partition, created it again as NTFS.

While formatting the new volume, I wanted to see what would happen if I do not choose Quick Format. Format took 5 hours for the 2 TB drive!

And after that, the backup of C, D and E drives, creating a 239.4 GB single file, took around 12 hours. 

Sunday, May 14, 2017

stopping unwanted notifications on Android

Some apps bring up annoying notifications. Examples - the activation messages for Cloud Messaging on Xiaomi phones, Airtel's and Paytm's "offers" and so on. An easy solution was to pull down the notification bar, click and hold the notification till a box appears with "App Info" - may be seen as an exclamation mark icon. Click on that, which gives an option to block notifications from that app. 

preparing Mandir Live Files for Download

From an email by PA.

Steps to process live files

1. Open finder and click on ‘Mandir Live Recordings’, click date modified

2. The first few files will be that morning's and the previous days

3. Open ‘Magic Cutter’, drag and drop the file with yesterday’s date. Morning file will end with ‘_1’ and evening with ‘_2’

4. Double clicking on the wave will give you the split marker. Place one marker just when ‘Omkar’ of vedam starts and one before ‘Omkar’ of bhajans start. (You will need to connect the earphones to the mac mini's 3.5 jack to hear the file)

5. Then click ‘Split’

6. Now it will ask for where to save. Click the drop down on top of the dialogue box and select the folder ‘Split Files’

7. It will take 2 seconds to complete. Then repeat the same for Evening file.

8. After both files have been split, close Magic Cutter. In the finder go to the folder ‘Split Files’ in the sidebar.

9. Once there, again click date modified. You will see 6 files with names different from the rest. Date followed by ’_01_01’ and so on.

10. Now delete the 2 files that have the announcement. (They will be the 2 files with least size)

11. Now rename the files just like the other files in the folder. Morning vedam and bhajan files should have the date and end with '_AM_VEDAM' and '_AM_BHAJAN' respectively
Evening vedam and bhajan files should end with '_PM_VEDAM' and '_PM_BHAJAN' respectively

12. After renaming, close Finder and open 'Filezilla'

13. In the menu right on top of the monitor, click bookmarks and select the bookmark, 'Split Files'.

14. Once the connection is established, transfer the files that you just renamed. Double-click files one by one, or select, drag & drop.

Friday, April 28, 2017

Blender links

Some Blender links which I found useful:

https://blender.stackexchange.com/questions/124/how-to-emulate-a-number-pad-and-3-button-mouse

https://blender.stackexchange.com/questions/644/why-does-the-zoom-sometimes-stop-at-a-point

https://www.youtube.com/watch?v=zmY_mn6Ue2g

In this tutorial, there is a mention of how the output may look grainy if the object in blender is not scaled up. But I was not successful in improving my OCT renders though I scaled up the cube 2 or 3 times.

Tuesday, April 25, 2017

installing the latest blender on Linux Mint

I needed to install Blender 2.78 for compatibility with files created on Windows with 2.78. The Linux Mint repository (Ubuntu repository) carried the earlier version of Blender. So, followed this guide to install 2.78.

sudo add-apt-repository ppa:thomas-schiex/blender
sudo apt update 
sudo apt install blender

To go back to the default version, I need to do
sudo apt install ppa-purge 
sudo ppa-purge ppa:thomas-schiex/blender

Subscribe to: Posts (Atom)