Sunday, November 05, 2017

more on SER, FITS and Octave

Adding to my earlier post - Siril had a stackall command which stacks all the SER files in a folder to sum-stacked FITS files. That would have taken away the drudgery of having to open 830 SER files manually one by one and converting them though the creation of directories could be automated by

mkdir $(seq --format 's%03.0f' 1 830)

(from
https://unix.stackexchange.com/questions/48750/creating-numerous-directories-using-mkdir )

which would create
s001
s002
etc upto
s830.

Unfortunately, Siril's stackall does sum stacking, and (at least according to my understanding, in version 0.9.7) was integer overflowing or showing some such behaviour - though the developers mentioned that it is supposed to convert to 64 bit, and then normalized to 16 bit. Maybe the normalizing was what was causing the issue. My data set had numbers like 44,000 per pixel and 102 frames, resulting in sums like 4,400,000 - even signed 32 bit int would only overflow at 2,147,483,647. Anyway, using the stackall sum stacking, I was getting plots like this

instead of plots like this.
Possible solution may be to implement a median stacking function for siril, like stackallmedian.


Friday, November 03, 2017

compiling QHYCCD camera SDK sample application with OpenCV

Steps needed to cleanly compile the LiveFrameSample code from QHY camera SDK, once OpenCV is installed -

Tested the opencv installation, compiling and running webcam capture test code from http://henryhermawan.blogspot.in/2007/09/capturing-frames-from-usb-camera-using.html

Followed the INSTALL file and compiled + installed the QHYCCD SDK.

cmake -DCMAKE_INSTALL_PREFIX=/usr .
make
sudo make install

Tried the sample without modifications -
cd sample
mkdir build
cd build
cmake ..
make

LiveFrameSampletest, SingleFrameSampletest and ControlCFWtest compiled without errors, but for CaptureDarkFrametest errors were reported, like
CaptureDarkFrame.cpp:76:83: error: invalid conversion from ‘int*’ to ‘uint32_t* {aka unsigned int*}’ [-fpermissive]
         ret = GetQHYCCDChipInfo(camhandle,&chipw,&chiph,&w,&h,&pixelw,&pixelh,&bpp);

and so on.

Anyway, I was only interested in the frame capture tests, so I created LiveFrameSampleOCV.cpp from LiveFrameSample.cpp and made the following changes -

  1. Uncommented the #define for opencv support and made that line
    #define OPENCV_SUPPORT 1
  2. Changed the set resolution line to
    ret = SetQHYCCDResolution(camhandle,320,240, 320, 240);
    (first two arguments after camhandle are dimensions, next two are position)
  3. Defined an int key before the display loop, and changed the display loop to an infinite loop -
    //ret = QHYCCD_ERROR;
    //while(ret != QHYCCD_SUCCESS)
    while(1)
  4. Added a break on escape key,
    key=cvWaitKey(30);
    if (key == 27)
    break;
  5. Modified the CMakeLists.txt to uncomment
    find_package(OpenCV REQUIRED) and
    include_directories($(OpenCV_INCLUDE_DIR))

    and also changed the link libraries line to
    target_link_libraries(${FILENAME1} -lqhy ${LIBUSB_1_LIBRARIES} ${OpenCV_LIBS}
    where the OpenCV_LIBS have been added. 
  6. Ran cmake .. from the build directory again, and then
    make LiveFrameSampleOCVtest
So now ./LiveFrameSampleOCVtest shows a live display of the camera at 320x240 resolution, around 20-24 fps. 

While running cmake, it shows
 -- Found libusb-1.0:
--  - Includes: /usr/include/libusb-1.0
--  - Libraries: /usr/lib/x86_64-linux-gnu/libusb-1.0.so

But could not build LiveFrameSample without using cmake - the gcc command needs some more work - this does not work -

gcc -Wall LiveFrameSampleOCV.cpp -o LiveFrameSampleOCV -l/usr/lib/x86_64-linux-gnu/ `pkg-config --cflags --libs opencv usb-1.0`



installing the latest OpenCV with Python bindings on Linux Mint 18.1

Followed the Ubuntu 16.04 tutorial by Adrian Rosebrock with some changes. Since my installation had python using anaconda, I used conda install instead of pip install. And since my first attempt at creating a virtual environment failed, I went ahead with the installation without a virtual environment. Most of the requirements were already installed. My cmake string also had a different path to the python executable due to anaconda.

sudo apt-get update 
sudo apt-get upgrade
sudo apt-get install build-essential cmake pkg-config
sudo apt-get install libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev 
sudo apt-get install libgtk-3-dev
sudo apt-get install libatlas-base-dev gfortran
sudo apt-get install python2.7-dev python3.5-dev

OpenCV stable version is now 3.3.1, so
 
wget -O opencv.zip https://github.com/Itseez/opencv/archive/3.3.1.zip
unzip opencv.zip
wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.3.1.zip
unzip opencv_contrib.zip
 
conda install numpy
 
cd opencv-3.1.0/
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D INSTALL_C_EXAMPLES=ON \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib-3.3.1/modules \
    -D PYTHON_EXECUTABLE=/home/mac/anaconda2/bin/python \
    -D BUILD_EXAMPLES=ON ..
 
cmake rapidly went through its checks, then I had to do the build,
 
make
 
which took around half an hour judging by the timestamps - I was doing other things.... and then finally 

sudo make install
sudo ldconfig
 
Verified by doing
 
ls -l /usr/local/lib/python2.7/site-packages/
 
which showed up cv2.so
 
and finally linked this cv2.so to the site-packages folder, in my case
 
cd ~/anaconda2/lib/python2.7/site-packages
ln -s /usr/local/lib/python2.7/site-packages/cv2.so cv2.so
 
Tested by opening a python shell, importing cv2 and checking its version,

python
 >>> import cv2
>>> cv2.__version__
'3.3.1'
>>>



 
 


Monday, October 30, 2017

SER, FITS and Octave

While capturing a series of images in an autocorrelation experiment using a QHY5L-II and Sharpcap, instead of capturing as a series of PNG files, captured as SER files by mistake. From the experience of trying to use that data, found the following.

  1. SER capture is faster than PNG sequence capture - PNG was capturing at 20 fps or so, but dropping frames after 80 frames or so, and was taking the last 20 frames at around 5 fps. SER was not dropping frames at all. All this on the Menlo systems Lenovo laptop. 
  2. Octave and Matlab don't open SER files without conversion, the definitive site for SER files seems to be http://www.grischa-hahn.homepage.t-online.de/astro/ser/
  3. After some back and forth on this thread, found that Siril can export SER to a sequence of FITS images without losing the 16-bit resolution. 
  4. Imagemagick can be used to convert the FITS images to 16-bit TIFF using
    mogrify -format tif *.fits
  5. Alternately, the fits octave package can be used to read the data into Octave from the FITS files. Although Octave uses Imagemagick to implement imread, imread doesn't seem to work with the FITS files from Siril, complaining
    error: Magick++ exception: Magick: Unexpected end-of-file (/media/mac/SAM2TB/Autocorr/2017-10-16/Capture/14_10_49/Capture_00002.fit) reported by coders/fits.c:370 (ReadFITSImage)
  6. In order to install the fits package, had to install its dependency, with
    sudo apt-get install libcfitsio*
    and then run octave as root,
    sudo octave
    and within octave, run
    pkg install -forge fits
  7. Reading using the fits package and imread give different results.
    imread returns uint16 rows x columns
    read_fits_image returns double columns x rows, upside down.
    So, for equivalence with imread, we need to use flipud.
    b=read_fits_image('im.fits')
    b=b';
    b=flipud(b);
    imagesc(b)

    gives the same numbers as
    a=imread('im.tif')
  8. identify Capture_0035.png
    can be used to check whether the png is 8-bit or 16-bit. 
  9. Installing ISIS and SER Player turned out to be dead-ends, since they did not convert SER to PNG without converting to 8-bit.
Edit - more on siril's stackall here

Saturday, October 28, 2017

removing malware/adware from Intex phone - trois

Even after my earlier posts about removing malware from Intex phone, one issue persisted. Occasionally, Rs. 1.50 would be deducted as SMS charges. At first I thought it was some sort of location finding behaviour by some app - as I noted, this deduction was noticed after I installed Uber and PayTM apps. But even after uninstalling those apps, the issue remained - deduction of Rs. 1.50 once in a week or so.

Finally I contacted my carrier Airtel via email - which seems to be the only way to reach their customer service department. They advised that I could Dial *121*51# (Toll Free) or 12181 (Toll Free) to check last 5 deductions which have happened in the last 72 hours. Using this information, I found that the SMS charges were to a number 9582943043. Googling, found that this is an "activation feature" on Intex phones!

https://together.jolla.com/question/139190/intex-aqua-fish-re-activation-sms/

Now I seem to have removed the offending Intex services - using Kingroot and root uninstaller - searched for sms in system apps, found SmsRegister - com.android.smsregister - uninstalled it. This seems to have solved the issue - no sms sent for the last one month. 

Friday, October 27, 2017

a tale of redirects

I added a second domain to Godaddy's deluxe hosting, using My Products -> Web Hosting -> Manage All ->  Manage (under primary domain name)  -> More -> Hosted Domains. The root folder for this was configured like /seconddomain, but when trying to access the second domain, was getting a redirect to
http://firstdomain.org/seconddomain

Godaddy support, over the course of 3 days, told me that this was due to some redirects in the .htaccess file. But they could not fix it for me. Finally when I tried editing the .htaccess file, found that the issues were due to some redirects meant for the CodeIgniter framework, redirecting html files to their php equivalents. The original .htaccess was
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?$1 [L]

In order to make it work with the second domain, I added some conditions, and made it

# BEGIN WordPress

RewriteBase /
# END WordPress

RewriteCond %{HTTP_HOST} ^www.seconddomain.org$ [NC]
RewriteRule ^(.*)$ http://seconddomain.org/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^seconddomain.org$ [NC]
rewritecond %{REQUEST_FILENAME} !-f
rewritecond %{REQUEST_FILENAME} !-d
rewriterule ^(.*)$ /index.html [L]

RewriteCond %{HTTP_HOST} ^www.firstdomain.org$ [NC]
RewriteRule ^(.*)$ http://firstdomain.org/$1 [L,R=301]

RewriteCond %{HTTP_HOST} ^firstdomain.org$ [NC]
rewritecond %{REQUEST_FILENAME} !-f
rewritecond %{REQUEST_FILENAME} !-d
rewriterule ^(.*)$ /index.php?$1 [L]

Now both sites firstdomain.org and seconddomain.org seem to be working fine. www.firstdomain.org redirects to firstdomain.org and similarly www.seconddomain.org redirects to seconddomain.org.

Sunday, October 08, 2017

removing duplicates

Found some duplicate filename entries in our schedule page remote database. PB found that these were due to some files with filenames beginning with a carriage return - ASCII 13 -

and he found there were 144 such duplicated files.


I found that these had been added in the early days - double digit index numbers - and so could be deleted - did

SELECT * FROM `name_of_table` where LEFT (fileName, 1) = '\r'

and then deleted all those 141 records using the phpmyadmin interface.