phplist bounce processing issue

Based on a complaint received on 19 Nov, checked the server logs and found that Gmail is starting to mark our mails as spam because we're daily sending mail to bouncing gmail accounts. 

PB went to the reconcile users page in phplist and looked at the process of unsubscribing users who have 15 and more bounces. (Found about 9,527 users) - but did not run it since this was not 15 consecutive bounces. Running the bounce processing manually, the bounce processing takes 3 hours, identifies the bounces, but does not unconfirm subscribers with 5 consecutive bounces. So, running bounce processing is not solving the problem. We needed to fix the bounce processing rules which are being applied.

PB found, checking for the location where the rules are defined and found that phplist is picking up rules from the database table. Table names: our_prefix_bounceregex and our_prefix_bounceregex_bounce. But those tables showed empty records. Did not find any web interface to add rules to these tables. Did not find a backup with entries. PB downloaded the sourcecode (version 2.10.10) from https://sourceforge.net/projects/phplist/files/phplist/ and checked the phplist.sql file that creates the database on a new installation. There too the records were empty.

(Edit: Found a discussion about this in the phplist manual here. )

Then, a quick and dirty fix implemented by PB on my request: in phplist web interface "view bounces" selected the message id 4189, got the list of email ids that had bounced for the message id 4189

mysql> select id,email,confirmed from our_prefix_user_user where id IN ( select user from our_prefix_user_message_bounce where message=4189);

- about 1147 records

Unconfirmed the users with the following sql update:

mysql> UPDATE our_prefix_user_user SET confirmed=0  WHERE id IN (SELECT user FROM our_prefix_user_message_bounce WHERE message=4189);

Query OK, 1146 rows affected (0.54 sec)

Rows matched: 1147  Changed: 1146  Warnings: 0

compiling siril

Following the dependencies listed at
https://free-astro.org/index.php/Siril:install#Dependencies
I did the following to install dependencies on Linux Mint 18.1 -

sudo apt-get install gtk3*
sudo apt-get install cfitsio*
sudo apt-get install fftw3*
sudo apt-get install gsl*
sudo apt-get install libconfig++*
sudo apt-get install libraw*
sudo apt-get install libffms2*
sudo apt-get install libtiff*
sudo apt-get install libjpeg*
sudo apt-get install libpng*

Then running autogen.sh complained -
./autogen.sh: intltoolize: not found


and for that I did
sudo apt-get install intltool
sudo apt-get install libgsl*

This was enough to build it using
./autogen.sh
 make
 sudo make install

Thought about doing a full-fledged fork and pull request as in
https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project/

For that, I needed to set the ssh key, as in this stackoverflow post,

cat ~/.ssh/id_rsa.pub
and copied the key to github -> settings -> ssh key

git clone git@github.com:my-account-name/Siril-0.9.git
cd Si*
git remote add upstream git@github.com:lock042/Siril-0.9.git
git checkout master
git pull upstream master && git push origin master
git checkout -b feature/stackallmedian

It turned out that this version had a bug where stackall would cause a segmentation fault. This was fixed in version v0.9.8-beta3 which also had a stackall median command, which did not do normalization by default, so my problem was solved and I did not have code it myself to do the git pull requests and so on.

more on SER, FITS and Octave

Adding to my earlier post - Siril had a stackall command which stacks all the SER files in a folder to sum-stacked FITS files. That would have taken away the drudgery of having to open 830 SER files manually one by one and converting them though the creation of directories could be automated by

mkdir $(seq --format 's%03.0f' 1 830)

(from
https://unix.stackexchange.com/questions/48750/creating-numerous-directories-using-mkdir )

which would create
s001
s002
etc upto
s830.

Unfortunately, Siril's stackall does sum stacking, and (at least according to my understanding, in version 0.9.7) was integer overflowing or showing some such behaviour - though the developers mentioned that it is supposed to convert to 64 bit, and then normalized to 16 bit. Maybe the normalizing was what was causing the issue. My data set had numbers like 44,000 per pixel and 102 frames, resulting in sums like 4,400,000 - even signed 32 bit int would only overflow at 2,147,483,647. Anyway, using the stackall sum stacking, I was getting plots like this

instead of plots like this.

Possible solution may be to implement a median stacking function for siril, like stackallmedian.

Edit: The siril team has now implemented median stacking using the command stackall median - and current branch has no normalization applied, so it works well for me :)

compiling QHYCCD camera SDK sample application with OpenCV

Steps needed to cleanly compile the LiveFrameSample code from QHY camera SDK, once OpenCV is installed -

Tested the opencv installation, compiling and running webcam capture test code from http://henryhermawan.blogspot.in/2007/09/capturing-frames-from-usb-camera-using.html

Followed the INSTALL file and compiled + installed the QHYCCD SDK.

cmake -DCMAKE_INSTALL_PREFIX=/usr .

make

sudo make install

Tried the sample without modifications -
cd sample
mkdir build
cd build
cmake ..
make

LiveFrameSampletest, SingleFrameSampletest and ControlCFWtest compiled without errors, but for CaptureDarkFrametest errors were reported, like
CaptureDarkFrame.cpp:76:83: error: invalid conversion from ‘int*’ to ‘uint32_t* {aka unsigned int*}’ [-fpermissive]
         ret = GetQHYCCDChipInfo(camhandle,&chipw,&chiph,&w,&h,&pixelw,&pixelh,&bpp);
and so on.

Anyway, I was only interested in the frame capture tests, so I created LiveFrameSampleOCV.cpp from LiveFrameSample.cpp and made the following changes -

1. Uncommented the #define for opencv support and made that line
   #define OPENCV_SUPPORT 1
2. Changed the set resolution line to
   ret = SetQHYCCDResolution(camhandle,320,240, 320, 240);
   (first two arguments after camhandle are dimensions, next two are position)
3. Defined an int key before the display loop, and changed the display loop to an infinite loop -
   //ret = QHYCCD_ERROR;
   //while(ret != QHYCCD_SUCCESS)
   while(1)
4. Added a break on escape key,
   key=cvWaitKey(30);
   if (key == 27)
   break;
5. Modified the CMakeLists.txt to uncomment
   find_package(OpenCV REQUIRED) and
   include_directories($(OpenCV_INCLUDE_DIR))
   and also changed the link libraries line to
   target_link_libraries(${FILENAME1} -lqhy ${LIBUSB_1_LIBRARIES} ${OpenCV_LIBS}
   where the OpenCV_LIBS have been added. 
6. Ran cmake .. from the build directory again, and then
   make LiveFrameSampleOCVtest

So now ./LiveFrameSampleOCVtest shows a live display of the camera at 320x240 resolution, around 20-24 fps. 

While running cmake, it shows
 -- Found libusb-1.0:
--  - Includes: /usr/include/libusb-1.0
--  - Libraries: /usr/lib/x86_64-linux-gnu/libusb-1.0.so

But could not build LiveFrameSample without using cmake - the gcc command needs some more work - this does not work -

gcc -Wall LiveFrameSampleOCV.cpp -o LiveFrameSampleOCV -l/usr/lib/x86_64-linux-gnu/ `pkg-config --cflags --libs opencv usb-1.0`

installing the latest OpenCV with Python bindings on Linux Mint 18.1

Followed the Ubuntu 16.04 tutorial by Adrian Rosebrock with some changes. Since my installation had python using anaconda, I used conda install instead of pip install. And since my first attempt at creating a virtual environment failed, I went ahead with the installation without a virtual environment. Most of the requirements were already installed. My cmake string also had a different path to the python executable due to anaconda.

sudo apt-get update 
sudo apt-get upgrade
sudo apt-get install build-essential cmake pkg-config
sudo apt-get install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev 
sudo apt-get install libgtk-3-dev
sudo apt-get install libatlas-base-dev gfortran
sudo apt-get install python2.7-dev python3.5-dev

OpenCV stable version is now 3.3.1, so

wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.3.1.zip

unzip opencv.zip

wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.3.1.zip

unzip opencv_contrib.zip

conda install numpy

cd opencv-3.1.0/

mkdir build

cd build

cmake -D CMAKE_BUILD_TYPE=RELEASE \

    -D CMAKE_INSTALL_PREFIX=/usr/local \

    -D INSTALL_PYTHON_EXAMPLES=ON \

    -D INSTALL_C_EXAMPLES=ON \

    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.3.1/modules \

    -D PYTHON_EXECUTABLE=/home/mac/anaconda2/bin/python \

    -D BUILD_EXAMPLES=ON ..

cmake rapidly went through its checks, then I had to do the build,

make

which took around half an hour judging by the timestamps - I was doing other things.... and then finally 

sudo make install

sudo ldconfig

Verified by doing

ls -l /usr/local/lib/python2.7/site-packages/

which showed up cv2.so

and finally linked this cv2.so to the site-packages folder, in my case

cd ~/anaconda2/lib/python2.7/site-packages

ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so

Tested by opening a python shell, importing cv2 and checking its version,

python

 >>> import cv2
>>> cv2.__version__
'3.3.1'
>>>

Edit: August 2019 - updated instructions for installing on Ubuntu 18.04 - 
https://www.pyimagesearch.com/2018/05/28/ubuntu-18-04-how-to-install-opencv/