Scramjet Proxy

Les données stockées par ces cookies nous permermettent de personnaliser le contenu des annonces, d'offrir des fonctionnalités relatives aux réseaux sociaux et d'analyser notre trafic. Nous partageons également certains cookies et des informations sur l'utilisation de notre site avec nos partenaires de médias sociaux, de publicité et d'analyse, qui peuvent combiner celles-ci avec d'autres informations que vous leur avez fournies ou qu'ils ont collectées lors de votre utilisation de leurs services. Nos partenaires sont Google et ses partenaires tiers.

Scramjet Proxy

If you are self-hosting the Scramjet Hub and need to configure the actual Proxy server (which sits in front of the Hub), you typically use Nginx or Traefik.

Here is a sample Nginx configuration for a Scramjet Hub setup:

server 
    listen 80;
    server_name scramjet.yourdomain.com;
location / 
    # Forward all traffic to the Scramjet Hub
    proxy_pass http://localhost:9000; # Default Scramjet Hub API port
# Standard Proxy Headers
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# WebSocket support (crucial for streaming data)
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

Note: The Scramjet Hub handles the internal routing to the containers; Nginx just passes the traffic to the Hub. scramjet proxy

Create proxy-stream.js:

const  DataStream  = require('scramjet');
const fs = require('fs');
const axios = require('axios');
// Load proxies into a reusable array (will cycle)
const proxyList = fs.readFileSync('proxies.txt', 'utf-8')
.split('\n')
.filter(Boolean);
let proxyIndex = 0;
// Function to get next proxy (round-robin)
const getNextProxy = () => 
const proxy = proxyList[proxyIndex % proxyList.length];
proxyIndex++;
return proxy;
; If you are self-hosting the Scramjet Hub and
// Create a stream of URLs to scrape
const urlStream = DataStream.from([
'https://httpbin.org/ip',
'https://httpbin.org/ip',
'https://httpbin.org/user-agent'
]);
// The actual Scramjet Proxy pipeline
urlStream
.setOptions( maxParallel: 5 ) // 5 concurrent requests
.map(async (url) => 
const proxyUrl = getNextProxy();
try 
const response = await axios.get(url, 
proxy: 
host: proxyUrl.split(':')[1].replace('//', ''),
port: proxyUrl.split(':')[2],
auth: 
username: proxyUrl.split('@')[0].split(':')[1].replace('//', ''),
password: proxyUrl.split('@')[0].split(':')[2]
,
timeout: 10000
);
return  url, data: response.data, proxy: proxyUrl, status: 'success' ;
 catch (error) 
return  url, error: error.message, proxy: proxyUrl, status: 'failed' ;
)
.each(result => console.log(JSON.stringify(result, null, 2)))
.run();

Run it: node proxy-stream.js

This is a basic example. A production Scramjet Proxy would use pipeline composition, retry streams, and health-check transforms.

Most proxy rotators use a simple round-robin algorithm. Scramjet Proxy uses transform streams. If a specific IP gets rate-limited, the stream automatically buffers that request, rotates the IP, and retries without crashing the main thread.

Building a basic Scramjet Proxy requires Node.js (v14+) and the Scramjet framework.