Scrumlaunch
Software Engineer

Description

We seek a highly skilled and motivated Full-stack Engineer, with solid .Net experience, willing to dive into web scraping to join our team. If you have a passion for unlocking data from the web and enjoy the challenge of reverse engineering complex websites, ingesting terabytes of data, and building scalable decentralized systems we want to hear from you.

As a Software Engineer, you will play a critical role in reverse engineering the structure and behavior of websites to extract data efficiently and effectively. Your expertise in full-stack engineering, data streaming, web scraping, reverse engineering, and data acquisition will contribute to our mission of collecting valuable information from the web.

Requirements

Must have skills:

- Proficiency in C# and .NET Core/ASP.NET Core;

- Knowledge of RESTful API and GraphQL development;

- Understanding of microservices architecture;

- Familiarity with background services (e.g., Hangfire, Azure Functions, AWS Lambda);

- Strong skills in JavaScript, TypeScript;

- Familiarity in AWS services (EC2, Lambda, S3, SQS, DynamoDB);

- Familiarity in unit testing (xUnit, NUnit, Moq);

- Familiarity with message queues (AWS SQS, RabbitMQ, Kafka);

- Familiarity in Integration testing with Postman/Insomnia;

- Experience working in Agile/Scrum teams;

- Proficiency with Git (GitHub);

- Strong problem-solving and analytical skills;

- Effective communication for cross-functional collaboration;

- Adaptability to new technologies and frameworks.

Responsibilities

- Reverse Engineering Web Structures: Analyze and reverse engineer websites to understand their structure, data sources, and interaction patterns;

- Web Scraping Development: Develop custom web scraping services and tools to extract data from websites, APIs, and other online sources;

- Web Scraping Support: Keep our web scraping services updated on any changes in the data source collected;

- Data Parsing and Transformation: Implement data parsing and transformation processes to clean and structure scraped data for analysis;

- Crawling and Data Acquisition: Create web crawler services that navigate websites, follow links, and collect data efficiently;

- Rate Limiting and Anti-Bot Measures: Overcome anti-bot measures and implement rate-limiting strategies to scrape data while respecting websites' terms of service;

- Data Quality Assurance: Implement data quality checks to ensure accuracy and consistency in scraped data;

- Monitoring and Scalability: Set up monitoring systems to track scraping activities and ensure scalability for large-scale data acquisition;

- Documentation: Maintain comprehensive documentation of scraping methods, scripts, and data sources.

Additional Skills

- NodeJs experience;

- Retail/e-commerce experience/alternative data;

- Familiarity with headless browsers for automation and testing;

- Knowledge of API integration;

- Data storage and database skills for storing and managing massive log data;

- Strong SQL skills.

Work conditions

We offer:

- Opportunities to work on diverse and challenging projects;

- Flexible working conditions;

- A supportive and collaborative team environment;

- 12 vacation days per year;

- 5 sick days per year;

- English classes;

- Access to psychologist support for mental well-being.