Integrate FAM pipeline into robottelo #14028

Griffin-Sullivan · 2024-02-09T19:49:27Z

Problem Statement

Current FAM misc pipeline has been broken for a couple releases. This is due to some version changes, but also we were not testing things as accurately as possible. For instance, we were cloning upstream and running the develop branch.

Solution

Integrate the FAM pipeline into robottelo. This now runs everything on the satellite against itself, so we know that the collection being packaged in Satellite is working correctly.

Note

The expectation is not that this will pass 100%. Getting all of the modules and roles to pass will take individual setup that needs to be done in robottelo, AAP, or upstream in the test playbook itself. For now I would like to see that the tests are executing as expected and the results show helpful output on failure (ie: the actual ansible tasking logs). After merging, we will need to evaluate as teams how to remediate the issues based on component and team.

Griffin-Sullivan · 2024-02-09T19:50:05Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles

Griffin-Sullivan · 2024-02-09T22:06:31Z

Not a terrible start (30/86 failed). You can see though that the failure output is not very helpful since it doesn't give you the full ansible play logs. I'm gonna research if there's a way to force Jenkins or pytest to show everything.

evgeni · 2024-02-12T13:15:58Z

tests/foreman/sys/test_fam.py

+    # Edit config file
+    config_file = f'{FAM_ROOT_DIR}/tests/test_playbooks/vars/server.yml'
+    module_target_sat.execute(
+        f'cp {FAM_ROOT_DIR}/tests/test_playbooks/vars/server.yml.example {config_file}'
+    )
+    module_target_sat.execute(
+        f'sed -i "s/foreman.example.com/{module_target_sat.hostname}/g" {config_file}'
+    )
+    module_target_sat.execute(
+        f'sed -i "s/rhsm_pool_id:.*/rhsm_pool_id: {settings.subscription.rhn_poolid}/g" {config_file}'
+    )
+    module_target_sat.execute(
+        f'''sed -i 's/rhsm_username:.*/rhsm_username: "{settings.subscription.rhn_username}"/g' {config_file}'''
+    )
+    module_target_sat.execute(
+        f'''sed -i 's|subscription_manifest_path:.*|subscription_manifest_path: "data/{module_sca_manifest.name}.zip"|g' {config_file}'''
+    )
+    module_target_sat.execute(
+        f'''sed -i 's/rhsm_password:.*/rhsm_password: "{settings.subscription.rhn_password}"/g' {config_file}'''
+    )


given the amount of changes we need to perform here, would it be easier/cleaner if we'd deploy this file (server.yml) from a template and completely ignore the example we ship in the package?

What would the process of deploying the file from a template look like?

No idea ;)
I was hoping there is just a way in robotello?

I've found

robottelo/robottelo/hosts.py

Lines 881 to 892 in 97d9117

def put(self, local_path, remote_path=None):

"""Put a local file to the broker virtual machine.

If local_path is a manifest object, write its contents to a temporary file

then continue with the upload.

"""

if 'utils.manifest' in str(local_path):

with NamedTemporaryFile(dir=robottelo_tmp_dir) as content_file:

content_file.write(local_path.content.read())

content_file.flush()

self.session.sftp_write(source=content_file.name, destination=remote_path)

else:

self.session.sftp_write(source=local_path, destination=remote_path)

or

robottelo/robottelo/hosts.py

Lines 1003 to 1018 in 97d9117

puppet_conf = (

'[main]\n'

'vardir = /opt/puppetlabs/puppet/cache\n'

'logdir = /var/log/puppetlabs/puppet\n'

'rundir = /var/run/puppetlabs\n'

'ssldir = /etc/puppetlabs/puppet/ssl\n'

'[agent]\n'

'pluginsync = true\n'

'report = true\n'

'ignoreschedules = true\n'

f'ca_server = {proxy_hostname}\n'

f'certname = {cert_name}\n'

'environment = production\n'

f'server = {proxy_hostname}\n'

)

self.execute(f'echo "{puppet_conf}" >> /etc/puppetlabs/puppet/puppet.conf')

We would still have to replace the hostname and manifest name during the test setup. For now I'd leave this as is even though it's ugly and we can change in the future.

evgeni · 2024-02-12T13:16:31Z

tests/foreman/sys/test_fam.py

+    )
+
+    # Handle color output formatting
+    module_target_sat.execute('export ANSIBLE_NOCOLOR=True && export PY_COLORS=0')


does .execute() always start a new ssh/shell process? if so, those exports won't do anything

I'm actually not sure. Good question for @JacobCallahan

Looking at https://github.com/SatelliteQE/broker/blob/master/broker/hosts.py, it seems the ssh session is kept alive (if possible).

I'd probably still move that to the actual test invocation below.

Griffin-Sullivan · 2024-02-12T14:47:25Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles

Griffin-Sullivan · 2024-02-12T16:00:41Z

Reran cause I realized I needed to add some more test_playbooks to run. This is now running the right amount of tests.

evgeni · 2024-02-13T09:37:12Z

tests/foreman/sys/test_fam.py

+    # Handle color output formatting
+    module_target_sat.execute('export ANSIBLE_NOCOLOR=True && export PY_COLORS=0')
+    module_target_sat.execute(
+        f"sed -i 's/pytest -vv/pytest -v --color=no /g' {FAM_ROOT_DIR}/Makefile"


you want -vv here, which will make it output all ansible output (as you wanted)

Seems like that didn't change the output for how Jenkins parsed the logs.

Griffin-Sullivan · 2024-02-13T17:31:51Z

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles

Satellite-QE · 2024-02-13T18:38:42Z

PRT Result

Build Number: 5705
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles
Test Result : =========== 41 failed, 69 passed, 138 warnings in 3719.12s (1:01:59) ===========

evgeni · 2024-02-14T09:36:02Z

tests/foreman/sys/test_fam.py

+
+    # Handle color output formatting
+    module_target_sat.execute(
+        f"sed -i 's/pytest -vv/pytest -vv --color=no /g' {FAM_ROOT_DIR}/Makefile"


setting PY_COLORS=0 below should be sufficient to force colors off. Sure this is required?

ok and I'm not sure. I never made those changes originally

evgeni · 2024-02-14T09:37:53Z

tests/foreman/sys/test_fam.py

+    """
+    # Execute test_playbook
+    result = module_target_sat.execute(
+        'export ANSIBLE_NOCOLOR=True && export PY_COLORS=0 && '


Both Ansible and Pytest support NO_COLOR so it should be sufficient to set NO_COLOR=True

evgeni · 2024-02-14T10:53:33Z

tests/foreman/sys/test_fam.py

+        'export ANSIBLE_NOCOLOR=True && export PY_COLORS=0 && '
+        f'. ~/localenv/bin/activate && cd {FAM_ROOT_DIR} && make livetest_{ansible_module}'
+    )
+    assert 'PASSED' in result.stdout


I think you want to also assert exit code being zero, just to be sure?

ok works for me

evgeni · 2024-02-14T11:07:18Z

I think I know why we're missing the "real" output in the test results.

The pytest that is running the "outer" tests is not called with -vv:

18:36:18  + pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles --external-logging --junit-xml=sat-results.xml -o junit_suite_name=sat-result

Griffin-Sullivan · 2024-02-14T15:52:05Z

I think I know why we're missing the "real" output in the test results.

The pytest that is running the "outer" tests is not called with -vv:
18:36:18  + pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles --external-logging --junit-xml=sat-results.xml -o junit_suite_name=sat-result

Interesting. I'll try changing it. Let me update everything and we'll rerun.

Griffin-Sullivan · 2024-02-14T16:00:17Z

trigger: test-robottelo
pytest: -vv tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles

Satellite-QE · 2024-02-14T17:13:29Z

PRT Result

Build Number: 5721
Build Status: UNSTABLE
PRT Comment: pytest -vv tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles
Test Result : =========== 41 failed, 69 passed, 137 warnings in 3850.63s (1:04:10) ===========

evgeni · 2024-02-14T17:46:20Z

Well, at least it didn't eat up the output now

Griffin-Sullivan · 2024-02-14T17:52:00Z

Yeah we can at least scroll to the bottom for the failure. Works for me.

JacobCallahan

great addition! just one minor thing

JacobCallahan · 2024-02-19T18:06:23Z

tests/foreman/sys/test_fam.py

+    module_target_sat.execute(f"sed -i '/^live/ s/$(MANIFEST)//' {FAM_ROOT_DIR}/Makefile")
+
+    # Upload manifest to test playbooks directory
+    module_target_sat.put(f'{module_sca_manifest.path}', f'{module_sca_manifest.name}')


why the f-strings here?

ogajduse

Co-reviewed with @jameerpathan111
LGTM, let's see how it behaves in the production.

(cherry picked from commit 7aca395)

Integrate FAM pipeline into robottelo (#14028) (cherry picked from commit 7aca395) Co-authored-by: Griffin Sullivan <48397354+Griffin-Sullivan@users.noreply.github.com>

* Integrate FAM pipeline into robottelo (#14028) (cherry picked from commit 7aca395) * Fix manifest file transer for fam --------- Co-authored-by: Griffin Sullivan <48397354+Griffin-Sullivan@users.noreply.github.com> Co-authored-by: Griffin-Sullivan <gsulliva@redhat.com>

Integrate FAM pipeline into robottelo (SatelliteQE#14028) (cherry picked from commit 7aca395) Co-authored-by: Griffin Sullivan <48397354+Griffin-Sullivan@users.noreply.github.com>

Griffin-Sullivan requested review from a team as code owners February 9, 2024 19:49

github-actions bot assigned Griffin-Sullivan Feb 9, 2024

evgeni reviewed Feb 12, 2024

View reviewed changes

Griffin-Sullivan force-pushed the fam branch from b7f1d3d to 2602e4e Compare February 12, 2024 14:47

evgeni reviewed Feb 13, 2024

View reviewed changes

Griffin-Sullivan force-pushed the fam branch from 2602e4e to 4e94038 Compare February 13, 2024 17:31

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Feb 13, 2024

evgeni reviewed Feb 14, 2024

View reviewed changes

Griffin-Sullivan force-pushed the fam branch from 4e94038 to 633573c Compare February 14, 2024 15:59

Griffin-Sullivan force-pushed the fam branch from 633573c to 92a1e0d Compare February 16, 2024 13:30

evgeni approved these changes Feb 16, 2024

View reviewed changes

JacobCallahan requested changes Feb 19, 2024

View reviewed changes

Integrate FAM pipeline into robottelo

3b2fd71

Griffin-Sullivan force-pushed the fam branch from 92a1e0d to 3b2fd71 Compare February 20, 2024 13:40

Griffin-Sullivan requested a review from JacobCallahan February 20, 2024 13:41

JacobCallahan approved these changes Feb 20, 2024

View reviewed changes

ogajduse approved these changes Feb 21, 2024

View reviewed changes

jameerpathan111 approved these changes Feb 21, 2024

View reviewed changes

jameerpathan111 merged commit 7aca395 into SatelliteQE:master Feb 21, 2024
6 checks passed

github-actions bot pushed a commit that referenced this pull request Feb 21, 2024

Integrate FAM pipeline into robottelo (#14028)

6102e9a

(cherry picked from commit 7aca395)

Satellite-QE mentioned this pull request Feb 21, 2024

[6.13.z] Integrate FAM pipeline into robottelo #14155

Closed

github-actions bot pushed a commit that referenced this pull request Feb 21, 2024

Integrate FAM pipeline into robottelo (#14028)

fc1f31a

(cherry picked from commit 7aca395)

Satellite-QE mentioned this pull request Feb 21, 2024

[6.15.z] Integrate FAM pipeline into robottelo #14156

Merged

github-actions bot pushed a commit that referenced this pull request Feb 21, 2024

Integrate FAM pipeline into robottelo (#14028)

616e995

(cherry picked from commit 7aca395)

Satellite-QE mentioned this pull request Feb 21, 2024

[6.14.z] Integrate FAM pipeline into robottelo #14157

Merged

shweta83 pushed a commit to shweta83/robottelo that referenced this pull request Apr 10, 2024

Integrate FAM pipeline into robottelo (SatelliteQE#14028)

b0e5769

vijaysawant mentioned this pull request Jul 2, 2024

Remove non-existing ansible modules #15549

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate FAM pipeline into robottelo #14028

Integrate FAM pipeline into robottelo #14028

Griffin-Sullivan commented Feb 9, 2024 •

edited

Loading

Griffin-Sullivan commented Feb 9, 2024

Griffin-Sullivan commented Feb 9, 2024

evgeni Feb 12, 2024

Griffin-Sullivan Feb 12, 2024

evgeni Feb 12, 2024

evgeni Feb 13, 2024

evgeni Feb 13, 2024

Griffin-Sullivan Feb 15, 2024

evgeni Feb 12, 2024

Griffin-Sullivan Feb 12, 2024

evgeni Feb 13, 2024

Griffin-Sullivan commented Feb 12, 2024

Griffin-Sullivan commented Feb 12, 2024

evgeni Feb 13, 2024

Griffin-Sullivan Feb 14, 2024

Griffin-Sullivan commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

evgeni Feb 14, 2024

Griffin-Sullivan Feb 14, 2024

evgeni Feb 14, 2024

evgeni Feb 14, 2024

Griffin-Sullivan Feb 14, 2024

evgeni commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

evgeni commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

JacobCallahan left a comment

JacobCallahan Feb 19, 2024

Griffin-Sullivan Feb 20, 2024

ogajduse left a comment

	def put(self, local_path, remote_path=None):
	"""Put a local file to the broker virtual machine.
	If local_path is a manifest object, write its contents to a temporary file
	then continue with the upload.
	"""
	if 'utils.manifest' in str(local_path):
	with NamedTemporaryFile(dir=robottelo_tmp_dir) as content_file:
	content_file.write(local_path.content.read())
	content_file.flush()
	self.session.sftp_write(source=content_file.name, destination=remote_path)
	else:
	self.session.sftp_write(source=local_path, destination=remote_path)

	puppet_conf = (
	'[main]\n'
	'vardir = /opt/puppetlabs/puppet/cache\n'
	'logdir = /var/log/puppetlabs/puppet\n'
	'rundir = /var/run/puppetlabs\n'
	'ssldir = /etc/puppetlabs/puppet/ssl\n'
	'[agent]\n'
	'pluginsync = true\n'
	'report = true\n'
	'ignoreschedules = true\n'
	f'ca_server = {proxy_hostname}\n'
	f'certname = {cert_name}\n'
	'environment = production\n'
	f'server = {proxy_hostname}\n'
	)
	self.execute(f'echo "{puppet_conf}" >> /etc/puppetlabs/puppet/puppet.conf')

Integrate FAM pipeline into robottelo #14028

Integrate FAM pipeline into robottelo #14028

Conversation

Griffin-Sullivan commented Feb 9, 2024 • edited Loading

Problem Statement

Solution

Note

Griffin-Sullivan commented Feb 9, 2024

Griffin-Sullivan commented Feb 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Griffin-Sullivan commented Feb 12, 2024

Griffin-Sullivan commented Feb 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Griffin-Sullivan commented Feb 13, 2024

Satellite-QE commented Feb 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evgeni commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

Satellite-QE commented Feb 14, 2024

evgeni commented Feb 14, 2024

Griffin-Sullivan commented Feb 14, 2024

JacobCallahan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ogajduse left a comment

Choose a reason for hiding this comment

Griffin-Sullivan commented Feb 9, 2024 •

edited

Loading