Difference between revisions of "CNMC bare-metal"

From CNM Wiki
Jump to: navigation, search
(Server monitoring)
 
(33 intermediate revisions by the same user not shown)
Line 1: Line 1:
The [[CNMC bare-metal]] (hereinafter, the ''Metal'') is the technology that supports the [[CNM Cyber]] project, [[Urdu Project]], and [[CNM Cyber]]'s project in Ukraine.
+
The [[CNMC bare-metal]] (hereinafter, the ''Metals'') is the technology that supports the [[CNMCyber]] project, [[Urdu Project]], and [[CNMCyber]]'s project in Ukraine.
  
  
 
==Needs==
 
==Needs==
The ''Metal's'' owner would like to be in business of workforce development. The ''Metal'' shall provide learners, who are employment aspirants, with laboratory environment that would include various end-user applications, as well as virtual machines (hereinafter, ''VM''s) and containers that host them. The end-user applications shall include [[AVideo]], [[GitLab]], [[HumHub]], [[MediaWiki]], [[Moodle]], [[Odoo]], [[Roundcube]], [[SuiteCRM]], and [[WordPress]] and some others. Through their hands-on training and internships, the learners shall possess those competencies and credentials that shall help these learners to land new jobs or advance in their current employment. These competencies and credentials shall range from bare-metal server to end-user application design, implementation, administration, and retirement.
+
The ''Metals<nowiki>'</nowiki>'' owner would like to be in business of workforce development. The ''Metals'' shall provide learners, who are employment aspirants, with laboratory environment that would include various end-user applications, as well as virtual machines (hereinafter, ''VM''s) and containers that host them. The end-user applications shall include [[AVideo]], [[GitLab]], [[HumHub]], [[MediaWiki]], [[Moodle]], [[Odoo]], [[Roundcube]], [[SuiteCRM]], and [[WordPress]] and some others. Through their hands-on training and internships, the learners shall possess those competencies and credentials that shall help these learners to land new jobs or advance in their current employment. These competencies and credentials shall range from bare-metal server to end-user application design, implementation, administration, and retirement.
  
 
===User stories===
 
===User stories===
Line 9: Line 9:
 
#As a training provider, I want to have authority to give a particular learner to access a ''VM'' that hosts a particular software package, so this learner will be able to explore, provision, modify, and restore the package.
 
#As a training provider, I want to have authority to give a particular learner to access a ''VM'' that hosts a particular software package, so this learner will be able to explore, provision, modify, and restore the package.
 
#As a learner, I want to access a ''VM'', which I am authorized to access, in order to explore, provision, modify, and restore the application that this ''VM'' hosts.
 
#As a learner, I want to access a ''VM'', which I am authorized to access, in order to explore, provision, modify, and restore the application that this ''VM'' hosts.
 +
#As an employment candidate, I want to show my skills and experience of working with software during employment interviews.
  
 
===Milestones===
 
===Milestones===
 
We plan to:
 
We plan to:
#Purchase the ''Metal'', setup [[RAID]], [[OpenZFS]] and [[ProxmoxVE]], as well as secure them.
+
#Re-design the ''Metals'', [[RAID]], [[OpenZFS]] and [[ProxmoxVE]], as well as secure them.
#Setup a high-availability secure cluster of [[WordPress]] instances on the purchased ''Metal''.
+
#Setup a high-availability secure cluster of [[WordPress]] instances on the ''Metals''.
#Setup a high-availability cluster of three bare-metal servers, including the main server that has been used in the Ukraine project and and the ''Metal''.
+
#Explore high-availability and Geocast options for the instances installed on the ''Metals'' and VPSes.
  
 
==Server layer==
 
==Server layer==
To support the project technologically, we created a platform with 3 VPS at contabo.com and one proxmox instance at hetzner.de in Germany. At the nest stage of the project, we may add one or two bare-metal servers in the same datacenter with the existing one. The additional server or servers shall be similar to the current one, probably, with smaller RAM. Later, we would also like to explore OVH servers.
+
To support the project technologically, we created a platform with 3 VPS at contabo.com and one proxmox instance at hetzner.de in Germany. At the nest stage of the project, we may add one or two bare-metal servers in the same datacenter with the existing one. We would like to explore how to test the purchased server or servers. The additional server or servers may or may not be similar to the current one, probably, with smaller RAM. Later, we would also like to explore OVH servers.
  
===Bare-metal===
+
===Virtualization===
The characteristics of the current bare-metal server are as follows:
+
[[ProxmoxVE]] is the only package under consideration; [[Debian]] shall be used as its [[OS]].
 +
 
 +
===Server security===
 +
We plan 5 measures to ensure the ''Metals'' security at the server layer:
 +
#Using iptables, ban everything what is not allowed
 +
#fail2ban to prevent brute-force attacks
 +
#Changing default ports, especially 22/SSH
 +
#Setting up the new SSH port to key based authentication so who have the key they can access.
 +
#Creating a whitelist of static IP addresses that are allowed to access the new SSH port
 +
One specialist also proposed to use Config Server Security and Firewall (CSF), so when anyone do wrong attempt with our server it will auto blocked on three wrong attempt, also we can block any country any location any ip any isp etc with firewall.
 +
 
 +
===Server-level HA===
 +
[[ProxmoxVE]] ships with its HA tool called [[HA-manager]]; however, it requires 3 bare-metal servers to be implemented. At this stage, we consider working with only one bare-metal server. Thus, we consider using [[HAProxy]] and/or, possibly, [[CDN]] between the existing bare-metal server and VPSes.
 +
 
 +
==Metal 1==
 +
 
 +
The characteristics of the first bare-metal server are as follows:
 
*Dedicated Root Server SB35
 
*Dedicated Root Server SB35
 
*Intel Core i7-3930
 
*Intel Core i7-3930
Line 30: Line 47:
 
One expert stated that using i7 processors have some disadvantages. Another expert replied that, indeed, i7 processors may have troubles with something like [[PCI passthrough]], but for this particular project at this particular stage, they are just fine.
 
One expert stated that using i7 processors have some disadvantages. Another expert replied that, indeed, i7 processors may have troubles with something like [[PCI passthrough]], but for this particular project at this particular stage, they are just fine.
  
===Disk-redundancy===
+
===Disk-redundancy 1===
*At the bare-metal layer, [[RAID]] shall be used -- decisions what level? and what implementation? are needed to be made.
+
[[ProxmoxVE]] ships with [[OpenZFS]], so, we consider using it strongly. Other implementations of software-based [[RAID]] at various levels shall also be discussed. We don't consider hardware-based solutions at this stage to avoid dependency on proprietary RAID cards.
*At the software layer, [[OpenZFS]] can be used.
+
 
 +
===IP addresses 1===
 +
On the main server, either local IP or private IP range with DHCP is used. Unless specific concerns arise, we plan to use 2 IPv4 addresses. We are also open to explore IPv6. If we use About ipv4 addresses, we need 5 ips: one for main server, two for gateway, three for any vps or container and four for wordpress vps, and 5 for anyother we need in future.
 +
 
 +
[[OpenVPN]]
 +
 
 +
===Server monitoring 1===
 +
To monitor the ''Metals'' performance at the server layer, we plan to deploy both:
 +
*[[Zabbix]]
 +
*[[Nagios]]
 +
 
 +
==Metal 2==
  
===Virtualization===
+
The characteristics of the second bare-metal server are as follows:
OS installation first, so it will be proxmox, so we can create some shared container or kvm based container there as per our usage.
+
*Dedicated Root Server
 +
* Intel Xeon E3-1245
 +
* 2x HDD SATA 3,0 TB Enterprise
 +
* 4x RAM 8192 MB DDR3 ECC
 +
* NIC 1 Gbit Intel 82574L
 +
* RAID Controller 4-Port SATA PCI-E LSI MegaRAID SAS 9260-4i
 +
* Location: FSN1 (Falkenstein/Vogtland, Germany) -- DC7
 +
* Rescue system (English)
  
===Server security===
+
===Disk-redundancy 2===
We plan 5 measures to ensure the ''Metal'' security at the server layer:
+
[[ProxmoxVE]] ships with [[OpenZFS]], so, we consider using it strongly. Other implementations of software-based [[RAID]] at various levels shall also be discussed. We don't consider hardware-based solutions at this stage to avoid dependency on proprietary RAID cards.
#Using iptables, ban everything what is not allowed
 
#fail2ban to prevent brute-force attacks
 
#Changing default ports, especially 22/SSH
 
#Setting up the new SSH port to key based authentication so who have the key they can access.
 
#Creating a whitelist of static IP addresses that are allowed to access the new SSH port
 
One specialist also proposed to use Config Server Security and Firewall (CSF), so when anyone do wrong attempt with our server it will auto blocked on three wrong attempt, also we can block any country any location any ip any isp etc with firewall.
 
  
===IP addresses===
+
===IP addresses 2===
 
On the main server, either local IP or private IP range with DHCP is used. Unless specific concerns arise, we plan to use 2 IPv4 addresses. We are also open to explore IPv6. If we use About ipv4 addresses, we need 5 ips: one for main server, two for gateway, three for any vps or container and four for wordpress vps, and 5 for anyother we need in future.
 
On the main server, either local IP or private IP range with DHCP is used. Unless specific concerns arise, we plan to use 2 IPv4 addresses. We are also open to explore IPv6. If we use About ipv4 addresses, we need 5 ips: one for main server, two for gateway, three for any vps or container and four for wordpress vps, and 5 for anyother we need in future.
  
===Server monitoring===
+
[[OpenVPN]]
To monitor the ''Metal'' performance at the server layer, we plan to deploy both:
 
*Zabbix
 
*Nagios
 
  
===Server-level HA===
+
===Server monitoring 2===
HAProxy or out-of-the-box tools of proxmox
+
To monitor the ''Metals'' performance at the server layer, we plan to deploy both:
 +
*[[Zabbix]]
 +
*[[Nagios]]
  
 
==Application layer==
 
==Application layer==
We would like to start with Wordpress. Then, we add MediaWiki, Moodle and the rest. Our main server and its WordPress instances were hacked a few times. This is a description of what happened with the server (it is in Russian though) -- https://pravka.bskol.com/ru/%D0%9E%D0%BF%D1%8B%D1%82%D0%BD%D1%8B_%D0%9F%D1%80%D0%BE%D0%B5%D0%BA%D1%82
+
Our main server and its WordPress instances were hacked a few times. This is a description of what happened with the server (it is in Russian though) -- https://pravka.bskol.com/ru/%D0%9E%D0%BF%D1%8B%D1%82%D0%BD%D1%8B_%D0%9F%D1%80%D0%BE%D0%B5%D0%BA%D1%82
 +
 
 +
===HA clusters===
 +
We plan to use [https://pve.proxmox.com/wiki/High_Availability_Cluster High_Availability_Cluster]
 +
 
 +
In addition to the bare-server, we plan to deploy one VPS in the US to extend the high-availability of its core applications. We cannot consider Cloudflare for the project because this would limit hands-on training opportunities for future students.
 +
 
 +
===Monitoring===
 +
 
 +
===Firewalls===
 +
 
 +
==End-user applications==
 +
We would like to start with Wordpress. Then, we add MediaWiki, Moodle and the rest.
  
 
===WordPress===
 
===WordPress===
Line 65: Line 105:
  
 
*WordPress -- VM or container model? we can use kvm based vms as well, but normally containers will be ok. we can use one vps for wordpress also.  
 
*WordPress -- VM or container model? we can use kvm based vms as well, but normally containers will be ok. we can use one vps for wordpress also.  
*What would we do for its security? For securety we can use antivirus + CSF firewall
+
*What would we do for its security? For securety we can use antivirus
  
scanners
+
===MediaWiki===
  
===App-level HA===
+
===Moodle===
We plan to use [https://pve.proxmox.com/wiki/High_Availability_Cluster High_Availability_Cluster]
 
  
In addition to the bare-server, we plan to deploy one VPS in the US to extend the high-availability of its core applications. We cannot consider Cloudflare for the project because this would limit hands-on training opportunities for future students.
+
==Development==

Latest revision as of 19:10, 15 March 2023

The CNMC bare-metal (hereinafter, the Metals) is the technology that supports the CNMCyber project, Urdu Project, and CNMCyber's project in Ukraine.


Needs

The Metals' owner would like to be in business of workforce development. The Metals shall provide learners, who are employment aspirants, with laboratory environment that would include various end-user applications, as well as virtual machines (hereinafter, VMs) and containers that host them. The end-user applications shall include AVideo, GitLab, HumHub, MediaWiki, Moodle, Odoo, Roundcube, SuiteCRM, and WordPress and some others. Through their hands-on training and internships, the learners shall possess those competencies and credentials that shall help these learners to land new jobs or advance in their current employment. These competencies and credentials shall range from bare-metal server to end-user application design, implementation, administration, and retirement.

User stories

  1. As a workforce developer, I need my learners to have hands-on training opportunities on those software packages that are in demand on the job market.
  2. As a training provider, I want to have authority to give a particular learner to access a VM that hosts a particular software package, so this learner will be able to explore, provision, modify, and restore the package.
  3. As a learner, I want to access a VM, which I am authorized to access, in order to explore, provision, modify, and restore the application that this VM hosts.
  4. As an employment candidate, I want to show my skills and experience of working with software during employment interviews.

Milestones

We plan to:

  1. Re-design the Metals, RAID, OpenZFS and ProxmoxVE, as well as secure them.
  2. Setup a high-availability secure cluster of WordPress instances on the Metals.
  3. Explore high-availability and Geocast options for the instances installed on the Metals and VPSes.

Server layer

To support the project technologically, we created a platform with 3 VPS at contabo.com and one proxmox instance at hetzner.de in Germany. At the nest stage of the project, we may add one or two bare-metal servers in the same datacenter with the existing one. We would like to explore how to test the purchased server or servers. The additional server or servers may or may not be similar to the current one, probably, with smaller RAM. Later, we would also like to explore OVH servers.

Virtualization

ProxmoxVE is the only package under consideration; Debian shall be used as its OS.

Server security

We plan 5 measures to ensure the Metals security at the server layer:

  1. Using iptables, ban everything what is not allowed
  2. fail2ban to prevent brute-force attacks
  3. Changing default ports, especially 22/SSH
  4. Setting up the new SSH port to key based authentication so who have the key they can access.
  5. Creating a whitelist of static IP addresses that are allowed to access the new SSH port

One specialist also proposed to use Config Server Security and Firewall (CSF), so when anyone do wrong attempt with our server it will auto blocked on three wrong attempt, also we can block any country any location any ip any isp etc with firewall.

Server-level HA

ProxmoxVE ships with its HA tool called HA-manager; however, it requires 3 bare-metal servers to be implemented. At this stage, we consider working with only one bare-metal server. Thus, we consider using HAProxy and/or, possibly, CDN between the existing bare-metal server and VPSes.

Metal 1

The characteristics of the first bare-metal server are as follows:

  • Dedicated Root Server SB35
  • Intel Core i7-3930
  • 2x HDD SATA 3,0 TB
  • 8x RAM 8192 MB DDR3
  • NIC 1 Gbit - Intel 82579LM
  • Location: FSN1 (Falkenstein/Vogtland, Germany) -- DC7
  • Rescue system (English)

One expert stated that using i7 processors have some disadvantages. Another expert replied that, indeed, i7 processors may have troubles with something like PCI passthrough, but for this particular project at this particular stage, they are just fine.

Disk-redundancy 1

ProxmoxVE ships with OpenZFS, so, we consider using it strongly. Other implementations of software-based RAID at various levels shall also be discussed. We don't consider hardware-based solutions at this stage to avoid dependency on proprietary RAID cards.

IP addresses 1

On the main server, either local IP or private IP range with DHCP is used. Unless specific concerns arise, we plan to use 2 IPv4 addresses. We are also open to explore IPv6. If we use About ipv4 addresses, we need 5 ips: one for main server, two for gateway, three for any vps or container and four for wordpress vps, and 5 for anyother we need in future.

OpenVPN

Server monitoring 1

To monitor the Metals performance at the server layer, we plan to deploy both:

Metal 2

The characteristics of the second bare-metal server are as follows:

  • Dedicated Root Server
  • Intel Xeon E3-1245
  • 2x HDD SATA 3,0 TB Enterprise
  • 4x RAM 8192 MB DDR3 ECC
  • NIC 1 Gbit Intel 82574L
  • RAID Controller 4-Port SATA PCI-E LSI MegaRAID SAS 9260-4i
  • Location: FSN1 (Falkenstein/Vogtland, Germany) -- DC7
  • Rescue system (English)

Disk-redundancy 2

ProxmoxVE ships with OpenZFS, so, we consider using it strongly. Other implementations of software-based RAID at various levels shall also be discussed. We don't consider hardware-based solutions at this stage to avoid dependency on proprietary RAID cards.

IP addresses 2

On the main server, either local IP or private IP range with DHCP is used. Unless specific concerns arise, we plan to use 2 IPv4 addresses. We are also open to explore IPv6. If we use About ipv4 addresses, we need 5 ips: one for main server, two for gateway, three for any vps or container and four for wordpress vps, and 5 for anyother we need in future.

OpenVPN

Server monitoring 2

To monitor the Metals performance at the server layer, we plan to deploy both:

Application layer

Our main server and its WordPress instances were hacked a few times. This is a description of what happened with the server (it is in Russian though) -- https://pravka.bskol.com/ru/%D0%9E%D0%BF%D1%8B%D1%82%D0%BD%D1%8B_%D0%9F%D1%80%D0%BE%D0%B5%D0%BA%D1%82

HA clusters

We plan to use High_Availability_Cluster

In addition to the bare-server, we plan to deploy one VPS in the US to extend the high-availability of its core applications. We cannot consider Cloudflare for the project because this would limit hands-on training opportunities for future students.

Monitoring

Firewalls

End-user applications

We would like to start with Wordpress. Then, we add MediaWiki, Moodle and the rest.

WordPress

core, plugin, theme cloudflare as web app firewall

  • WordPress -- VM or container model? we can use kvm based vms as well, but normally containers will be ok. we can use one vps for wordpress also.
  • What would we do for its security? For securety we can use antivirus

MediaWiki

Moodle

Development