How I scrape data from my bank programmatically

---

TL;DR, you can use a text-based web browser to log into your bank account and scrape the data you need. This does not work with all banks but it does for some of the ones I use. For others, you can look at scraping an inbox that receives payment notifications from the bank.

If you send out invoices and receive EFT payments, chances are you have a process where you:

• Log in on your bank account
• Look if funds have arrived
• Log into your billing management solution (WHMCS, Quickbooks or perhaps you just have a spreadsheet)
• Mark the relevant invoice as paid
• Perform the necessary actions as required by the invoice (ship a product, register a domain, complete a service etc)

This might not be a big deal for a company just starting out who is receiving only a few payments a week but the moment you start receiving multiple payments per day this can turn into a time consuming process. Furthermore, if you only log in once a day, clients can get frustrated if they have to wait for you to action their payment.

A better solution would be to receive notifications from your bank or use an API that ties directly into your bank. However, very few banks offer this at an affordable price, at least that's the case in South Africa and Namibia. So failing that, the only solution is to try and automate the process I described above. 

How I automate logging into my bank account

This is briefly how I log into my bank account, grab the relevant information and store it locally:

1. Firstly, install Lynx
2. Now you can run it from the command line in "Recording" mode by doing this:
    lynx https://www.url-to-your-bank.com -cmd_log=mytest.keys -accept_all_cookies
   This will record all the keys you press and store it in mytest.keys.
3. Once you run this, you must navigate your banking portal to whatever page is showing your transactions. From here, your next steps will depend on what's available:
    - First prize would be to download the latest transactions as CSV and store it locally.
    - Alternatively, you can try and download other file formats or parse the screen display
4. Once you have recorded your keys, you can then again execute the same instructions like so:
    lynx -cmd_script=mytest.keys https://www.url-to-your-bank.com  2>&1
5. By executing the script repeatedly, you will generate a new file locally containing the relevant transactions you want

Please do note:

• This method might be against the terms and conditions of your Bank. Please make sure you are allowed to do this by asking them first. 
• Some banks will block you even if you are technically able to use this method, so again, it might not work for all cases.
• Lynx is very old and can't access a big variety of websites. It might be worthwhile looking at other text-based browsers, such as Browsh.
• If your bank sends out email notifications, a better alternative might be to rather parse those emails.