COMbination of Bipolar Androgen Therapy and Nivolumab

Inclusion Criteria: * Willing and able to provide signed informed consent. * Males aged 18 years of age and above. * Histological or cytologic proof of adenocarcinoma of the prostate. * Known castration-resistant disease, defined according to Prostate Cancer Working Group 3 (PCWG3) criteria as: * Castrate serum testosterone level: ≤ 50 ng/dL (≤ 1.7 nmol/L). * Subjects who have failed initial hormonal therapy, either by orchiectomy or by using a gonadotropin-releasing hormone (GnRH) agonist in combination with an anti-androgen, must first progress through anti-androgen withdrawal prior to being eligible. The minimum time frame to document failure of anti-androgen withdrawal will be four weeks. * Serum Prostate Specific Antigen (PSA) progression defined as two consecutive increases in PSA over a previous reference value within 6 months of first study treatment, each measurement at least one week apart. Or * Documented bone lesions by the appearance of ≥ 2 new lesions by bone scintigraphy or dimensionally-measureable soft tissue metastatic lesion assessed by CT or MRI. * Absolute PSA ≥ 2.0 ng/mL at screening. * Must have PSA and/or radiographic progression on AT LEAST 1 novel AR targeted therapy (abiraterone acetate, enzalutamide). One prior chemotherapy agent for metastatic castration-resistant prostate cancer (mCRPC) will be allowed. * Prior treatment with abiraterone, enzalutamide, bicalutamide, and/or ketoconazole is allowed. There is no limit on the maximum number or types of prior hormonal therapies received. * Must be maintained on a GnRH analogue or have undergone orchiectomy. * Radiographic evidence of metastatic disease by CT scan and bone scan, performed within the prior 4 weeks. * Must have a soft tissue metastatic lesion available for biopsy collection to perform tumor tissue analysis. * Karnofsky Performance Status (KPS): ≥ 70% within 14 days before start of study treatment (ECOG ≤ 2). * Participants must have normal organ and bone marrow function measured within 28 days prior to administration of study treatment as defined below: * Hemoglobin ≥ 9.0 g/dL with no blood transfusion in the past 28 days. * Absolute neutrophil count (ANC) ≥ 1.0 x 10\^9/L * Platelet count ≥ 100 x 10\^9/L * Total bilirubin within institutional upper limit of normal (ULN) (In patients with Gilbert's syndrome, total bilirubin \< 1.5x institutional ULN will be acceptable). * Aspartate aminotransferase (AST) (Serum Glutamic Oxaloacetic Transaminase (SGOT)) / Alanine aminotransferase (ALT) (Serum Glutamic Pyruvate Transaminase (SGPT)) within institutional upper limit of normal. * Participants must have creatinine clearance estimated ≥ 40 mL/min. * Participants must have a life expectancy ≥ 6 months. * Male participants and their partners, who are sexually active and of childbearing potential, must agree to the use of two highly effective forms of contraception in combination, throughout the period of taking study treatment and for 7 months after the last dose of nivolumab to prevent pregnancy in a partner. * No evidence (within 5 years) of prior malignancies (except successfully treated basal cell or squamous cell carcinoma of the skin). Exclusion Criteria: * Has received external-beam radiotherapy within the last 2 weeks prior to start of study treatment. * Prior oral anti-androgen (e.g. bicalutamide, nilutamide, enzalutamide, apalutamide), or androgen synthesis inhibitor (e.g. abiraterone, orteronel) within the past 2 weeks is not permitted. 5-alpha reductase inhibitor therapy (e.g. finasteride, dutasteride) is allowed, as long as subject has been stable on medication for past 6 months. * Prior treatment with chemotherapy for the treatment of metastatic hormone sensitive prostate cancer is allowed if the last dose of chemotherapy was greater than 6 months prior to enrollment. In addition, one chemotherapy agent for mCRPC will be allowed. * Patients who have received prior treatment with bipolar androgen therapy (e.g. testosterone, BAT) are excluded. * Pain due to metastatic prostate cancer requiring opioid therapy. * Patients with an intact prostate AND urinary obstructive symptoms are excluded (which includes patients with urinary symptoms from benign prostatic hyperplasia (BPH)). * Patients receiving anticoagulation therapy with Coumadin are not eligible for study. (Patients on non-coumadin anticoagulants (Lovenox, Xarelto, etc.) are eligible for study. Patients on Coumadin who can be transitioned to Lovenox or Xarelto prior to starting study treatments will be eligible). * Patients with prior history of an arteriovenous thromboembolic event within the last 12 months are excluded. * Patients allergic to sesame seed oil or cottonseed oil are excluded. * Involvement in the planning and/or conduct of the study (applies to both BMS staff and/or staff at the study site). * Participation in another clinical study with an investigational product during the last 4 weeks/28 days. * Patients should be excluded if they have had prior systemic treatment with an anti-Programmed Cell Death Protein (PD)-1, anti-PD-L1, anti-PD-L2, anti-Cytotoxic T-lymphocyte-Associated Protein (CTLA)-4 antibody, or any other antibody or drug specifically targeting T-cell costimulation or immune checkpoint pathways (e.g. immune checkpoint antagonists). * Evidence of disease in sites or extent that, in the opinion of the investigator, would put the patient at risk from therapy with testosterone (e.g. femoral metastases with concern over fracture risk, severe and extensive spinal metastases with concern over spinal cord compression, extensive liver metastases). * Concurrent use of other anticancer agents or treatments, with the following exceptions: * Ongoing treatment with leutinizing hormone-releasing hormone (LHRH) agonists or antagonists, denosumab (Prolia) or bisphosphonate (e.g. zoledronic acid) is allowed. Ongoing treatment should be kept at a stable schedule; however, if medically required, a change of dose, compound, or both is allowed. * Any treatment modalities involving major surgery within 4 weeks prior to the start of study treatment. * Symptomatic nodal disease, i.e. scrotal, penile or leg edema (≥ CTCAE Grade 3). * Patients are excluded if they have active known brain metastases or leptomeningeal metastases. Subjects with brain metastases are eligible if metastases have been treated and there is no magnetic resonance imaging (MRI) evidence of progression for at least 4 weeks after treatment is complete and within 28 days prior to the first dose of testosterone administration. There must also be no requirement for immunosuppressive doses of systemic corticosteroids (\> 10 mg/day prednisone equivalents) for at least 2 weeks prior to study drug administration. * Patients should be excluded if they have an active, known or suspected autoimmune disease (e.g. inflammatory bowel disease, rheumatoid arthritis, autoimmune hepatitis, lupus, celiac disease). Subjects are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger. * Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration. Inhaled or topical steroids and adrenal replacement doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease. * Permitted therapies include topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption). Physiologic replacement doses of systemic corticosteroids are permitted, even if \> 10 mg/day prednisone equivalents. A brief course of corticosteroids for prophylaxis (e.g. contrast dye allergy) or for treatment of non-autoimmune conditions (e.g. delayed-type hypersensitivity reaction caused by contact allergen) is permitted. * As there is potential for hepatic toxicity with nivolumab, drugs with a predisposition to hepatotoxicity should be used with caution in patients treated with nivolumab-containing regimen. * Patients should be excluded if they have a positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (HCV antibody) indicating acute or chronic infection. * Patients should be excluded if they have known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS). * History of allergy to study drug components. * History of severe hypersensitivity reaction to any monoclonal antibody. * Other primary tumor (other than CRPC) including hematological malignancy present within the last 5 years (except non-melanoma skin cancer or low-grade superficial bladder cancer). * Has imminent or established spinal cord compression based on clinical findings and/or MRI. * Any other serious illness or medical condition that would, in the opinion of the investigator, make this protocol unreasonably hazardous, including, but not limited to: * Any uncontrolled major infection. * Cardiac failure New York Heart Association (NYHA) Class III or IV. * Crohn's disease or ulcerative colitis. * Bone marrow dysplasia. * Known allergy to any of the compounds under investigation. * Unmanageable fecal incontinence. * Persistent toxicities (\> CTCAE Grade 2) caused by previous cancer therapy, excluding alopecia. * Poor medical risk due to a serious, uncontrolled medical disorder, non-malignant systemic disease or active, uncontrolled infection. Examples include, but are not limited to, uncontrolled ventricular arrhythmia, recent (within 6 months) myocardial infarction, uncontrolled major seizure disorder, extensive interstitial bilateral lung disease, or any psychiatric disorder that prohibits obtaining informed consent.