Use puppeteer on the server in non-headless mode

๐Ÿ““๐Ÿ““ TL:DR
Using puppeteer on the server is necessary use case when you want to
  • Crawl millions of web pages of a certain URL
  • Perform Web Automation
  • Automate the testing of web apps
  • Implement Bot-like activities on web sites etc.
  • But, using it in non - headless mode is such a pain. Especially, when you have any Unix-like OS i.e. ubuntu, centos installed on the remote machine and you are using the machine over SSH or any other remote connection tool like putty.
    ๐Ÿ›  How to do it :
    So, in spite of all these extreme cases we can still use it and in this article we will take a look at one of the ways to implement it.
    ๐Ÿ•ถ Some Background :
    Puppeteer at it's core uses the chromium browser, which does all the heavy lifting stuff like
  • Exposing DOM API's for interactions
  • Perform DOM manipulations and code injection
  • Cookie management and session handling
  • Navigation and context maintenance etc.
  • Chromium needs the display adapters to launch the chromium window on the host machine. Regardless of the host OS, it uses the available display adapter API available.
    When we are serving puppeteer using virtual machine or any other remote machine, by default we don't have any displays because all the work is done by Bash (AKA command line).
    If we have windows server, then there is no big deal. Just connect the server using Remote Desktop Connection and you are good to go. But, if you have Ubuntu, centos or any other UNIX based OS installed on the remote machine, then you have to do some extra efforts.
    ๐Ÿ‘‰ Steps
    โš’ 1. Regular stpes :-
    Before doing anything else, lets update host by some OLD school stuff
    sudo apt-get update
    
    sudo apt-get upgrade
    2. ๐Ÿ–ฅ Install desktop-like enviroments
    For installing the desktop based utility known as XRDP, I have installed core utilities like ubuntu-mate-core and ubuntu-mate-desktop by using this
    sudo apt-get install --no-install-recommends ubuntu-mate-core ubuntu-mate-desktop -y
    Ubuntu Mate Core :-
    This utility converts the basic Ubuntu machine to complete workstation by adding some extra applications.
    Ubuntu Mate Desktop :-
    This is on such desktop like environments for Ubuntu. This exposes the interface for handling the local as well as networked files, perform calculations etc.
    3. ๐Ÿ’ป Install XRDP :-
    After doing above steps, we have to install XRDP server which will give us the access of Remote Desktop Connection for host machine. We will also install some secondary dependencies for implementing complete desktop like environment.
    sudo apt-get install mate-core mate-desktop-environment mate-notification-daemon xrdp -y
    ๐Ÿ‘ฎ 4. Create remote user :-
    As suggested by numerous manuals, we will now create non-root user for performing all of our remaining tasks.
    Create a XYZ user and grant the sudo access to newly created user for avoiding any permission conflicts.
    adduser xseven
    
    usermod -aG admin xseven
    
    usermod -aG sudo xseven
    
    su - xseven
    ๐Ÿ”ง 5. Configuring Mate Session :
    We now will configure the necessary Mate Session. This will help use to customize the available desktop like enviroment as per our needs
    echo mate-session> ~/.xsession
    
    sudo cp /home/xseven/.xsession /etc/skel
    ๐Ÿš€๐Ÿš€๐Ÿš€ 6. Restart XRDP :-
    Final step is to restart XRDP to apply all the configrations.
    sudo service xrdp restart
    Once restarted, we can connect to the remote machine by using RDC or any other compatible tool
    This is how we can run puppteer in non-headless mode on server. There are many other alternatives like
  • Installing GNome or KDE for desktop machines
  • Host applications on desktop ubuntu enviroment
  • etc.
    Thanks For Reading....

    37

    This website collects cookies to deliver better user experience

    Use puppeteer on the server in non-headless mode