Sunday, February 05, 2023

exporting mysql table to csv

I wanted to export quite a large subset of a table which was using up 1.8 GB - first tried this method,
https://phoenixnap.com/kb/mysql-export-table-to-csv

 SELECT ourfield1, ourfield2, ourfield3 FROM ourTable
INTO OUTFILE '/path/to/ExportFile.csv'
FIELDS ENCLOSED BY '"'
TERMINATED BY ';'
ESCAPED BY '"'
LINES TERMINATED BY '\r\n'
;

This failed, --secure-file-priv is set to null(?) on our server and I didn't want to change that.

database - How should I resolve --secure-file-priv in MySQL? - Stack Overflow

Next plan was to export by redirecting a here document in a bash script. One hiccup was that the here doc should end with the end characters without any indent, and there should be no other characters after it on its line, otherwise the script fails to find the characters.

https://stackoverflow.com/questions/18660798/here-document-gives-unexpected-end-of-file-error 

Next issue was that the exported data was large, and probably not indexed, so
mysql: Out of memory (Needed 698144116 bytes)

For this, I would probably need to limit the number of result rows (by timestamp, in this case),

How do I get the current Unix time in milliseconds in Bash? - Server Fault



How to increment a variable in bash? - Ask Ubuntu

Then, the easiest way to code seemed to be to put the entire statement in one line, avoiding issues with SQL/bash multi-line statements:

$MYSQL --batch -u $MyUSER -h $MyHOST -p$MyPASS $DBS -e "select l.id, l.timecreated, l.userid, l.ip, f.filename from prefix_logstore_standard_log l left join prefix_files f on l.contextid = f.contextid where l.timecreated > $StartDate  and l.timecreated < $EndDate and l.action = 'viewed' and l.target='course_module' and f.filename like '%.pp%' order by l.timecreated;" > $FILE

This would actually give a tab separated file and not comma separated, but this is also imported without issues by Google Sheets.

 

No comments:

Post a Comment