Automating Solar Panel Price Comparison Data Updates with GitHub Actions

[zaryabahmedd]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

General

Discussion Details

Hi everyone,

I'm working on a solar energy information website where we regularly update solar panel pricing data, specifications, and market trends for consumers in Pakistan.

Currently, we're manually updating our price comparison tables and product listings, which is time-consuming. I'm exploring how to use GitHub Actions to automate parts of this workflow.
What I'm trying to accomplish:

Automatically fetch and update solar panel pricing data from various suppliers
Schedule weekly price comparison updates
Auto-generate markdown files with updated product specifications
Deploy changes to our static site automatically

My questions:

What's the best approach to schedule data scraping/API calls using GitHub Actions workflows?
How can I store and version control pricing data efficiently in the repository?
Are there any rate limiting concerns I should be aware of when running frequent automated updates?
Has anyone built similar price comparison automation workflows?

I'm relatively new to CI/CD pipelines, so any guidance, workflow examples, or best practices would be greatly appreciated!
Thanks in advance for your help!

[deadlinecode]

While GitHub Action Pipelines can be used for this i would recommend running something like a cronjob on your server (if you have a server that you also have shell access to or similar). Not only because its easier but also more secure. However this is totally up to you and depends on the infrastructure you have setup for your website. But if you have a typical setup where you have one server that hosts your website + maybe a database to store prices or even just the HTML for your website most of the time its easier and more secure to just have a cronjob running on that server that fetches and calculates the data and directly places it into the database or the HTML files.

Otherwise you would have to setup some type of access for GitHub pipelines since they run on GitHubs own servers and need a way to upload or update data or files on your server.

There is also the option of hosting your own GitHub pipeline on your server but i think this would be a little bit overkill / too complicated for your use case.

Also note that you have only 2.000 free runner minutes per month after that you will need to pay GitHub for running your pipelines.

If you need any more guidance on how to start with anything of what i mentioned feel free to ask :)