Nivolumab (Anti-PD1), Tadalafil and Oral Vancomycin in People With Refractory Primary Hepatocellular Carcinoma or Liver Dominant Metastatic Cancer From Colorectal or Pancreatic Cancers

* INCLUSION CRITERIA: * Patients must have * histopathological confirmation of Hepatocellular Carcinoma (HCC) (Cohort 1) OR * histopathological confirmation of carcinoma by in the setting of clinical and radiological characteristics which, together with the pathology, are highly suggestive of a diagnosis of HCC (Cohort 1) OR * histopathological confirmation of advanced colorectal or pancreatic malignancy with liver involvement as dominant site of metastasis (Per multidiscipline tumor board review and approval) (Cohort 2). * Patients must have disease that is not amenable to potentially curative resection, transplantation or ablation. * Patients must have progressed on, been intolerant to, or refused prior sorafenib/lenvatinib and/or atezolizumab/bevacizumab therapy (Cohort 1 only). * Subjects must have progressed on or after standard systemic chemotherapy (at least one line of chemotherapy for patients with liver metastasis from pancreatic ductal adenocarcinoma (PDAC), at least two lines of chemotherapy for patients with liver metastasis from colorectal cancer (CRC) (Cohort 2 only). * Patients must have evaluable or measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 * Patients must have lesion accessible for biopsy and be willing to undergo pre- and posttreatment biopsies. * Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1 * If liver cirrhosis is present, patient must have a Child-Pugh score less than or equal to 7 * Active chronic hepatitis B virus (HBV) infected subjects must be on antivirals and have HBV deoxyribonucleic acid (DNA) \<100IU/mL. hepatitis C virus (HCV) infected subjects can be enrolled with close HCV RNA level monitoring. * Age greater than or equal to 18 years. Because no dosing or adverse event data are currently available on the use of nivolumab in combination with tadalafil and vancomycin in patients less than 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Adequate hematological function defined by: * white blood cell (WBC) count greater than or equal to 3x10\^9/L * absolute neutrophil count (ANC) greater than or equal to 1.5x10\^9/L, * lymphocyte count greater than or equal to 0.5x10\^9/L, * platelet count greater than or equal to 60x10\^9/L, and * Hemoglobin (Hgb) greater than or equal to 9 g/ dL (more than 48 hours post-completion of blood transfusion) * Adequate hepatic function defined by: * a total bilirubin level less than or equal to 1.5 X Upper limit of normal (ULN), * an aspartate transaminase (AST) level \<5xULN, * an alanine aminotransferase (ALT) level \<5xULN. * Adequate renal function defined by: * Creatinine clearance (CrCl) greater than or equal to 50 mL/min/1.73 m\^2 by 24 hours urine collection or as predicted by the Cockcroft-Gault formula: * CrCl = (140 age (y)) x (weight in kg) x (0.85, if female) x1.73 m\^2/72 x Serum Creatinine (mg/dL) x patient's body surface area (BSA) (m\^2) Or * The effects of nivolumab on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation and up to 5 months (women) and 7 months (men) after the last dose of the drug. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. * Patients with a history of cardiovascular disease may be enrolled per cardiology consultation and approval with echocardiogram and troponin level in normal range at the time of enrollment. * Patient must be able to understand and willing to sign a written informed consent document. EXCLUSION CRITERIA: * Patients who have had standard-of-care anti-cancer therapy or therapy with investigational agents (e.g., chemotherapy, immunotherapy, endocrine therapy, targeted therapy, biologic therapy, tumor embolization, monoclonal antibodies or other investigation agents), large field radiotherapy, or major surgery within 4 weeks prior to enrollment. * Therapy with antibiotics within 30 days prior to enrollment. * Therapy with nitrates, alpha-blockers, or cytochrome P450 (CYP3A4) inhibitors within 7- days prior to enrollment and for whom stopping is unsafe and/or a safe substitute is not medically recommended. Use of PDE5 inhibitors such as vardenafil (Levitra), tadalafil (Cialis), and sildenafil citrate (Viagra) less than or equal to 15-days prior to enrollment. * The patient must not be currently on a corticosteroid dose greater than physiologic replacement dosing defined as 10 mg of cortisone per day or its equivalent. * For PDAC patients with liver metastases, primary PDAC has not been resected (unless the primary is in the tail of the pancreas). * Patients with known brain metastases will be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events. * Have signs of liver failure, e.g., clinical significant ascites, encephalopathy, or variceal bleeding within 6 months prior to enrollment. * Prior major liver resection: remnant liver \<50% of the initial liver volume. Patients with a biliary stent can be included. * Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids. These include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome or Chronic Inflammatory Demyelinating Polyneuropathy (CIDP), myasthenia gravis; systemic autoimmune disease such as Systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome. Such diseases should be excluded because of the risk of recurrence or exacerbation of disease. Of note, patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible. Patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible. * Have history of idiopathic pulmonary fibrosis (including bronchiolitis obliterans with organizing pneumonia) or evidence of active pneumonitis on screening chest computed tomography (CT) scan. * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. * Patients with myocardial infarction or myocarditis within 12 months prior to enrollment. * History of severe or unstable cerebrovascular disease. * Sustained hypotension (\<90/50 mmHg) or uncontrolled hypertension (\>160/100 mmHg) * Stroke within 6 months prior to enrollment. * Human immunodeficiency virus (HIV)-positive patients are excluded because HIV causes complicated immune deficiency and study treatment can possess more risks for these patients. * Have had prior transplant of any kind. * Have ascites. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab, tadalafil or vancomycin. * History of severe hypersensitivity reaction to any monoclonal antibody. * Prior invasive malignancy (except non-melanomatous skin cancer) unless disease free for a minimum of 3 years prior to enrollment. * Pregnant women are excluded from this study because nivolumabs potential for teratogenic or abortifacient effects is unknown. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with nivolumab, breastfeeding should be discontinued if the mother is treated with nivolumab