N-803 Alone or in Combination With BN-Brachyury Vaccine or Bintrafusp Alfa (M7824) for Participants With Castration Resistant Prostate Cancer

* INCLUSION CRITERIA: * History of castration-resistant prostate cancer (CRPC) with histologic confirmation of diagnosis at any time point prior to initiation of study therapy. Note: If a pathology report is unavailable, a history consistent with prostate cancer is sufficient. * Prior treatment with testosterone lowering therapy for CRPC is required. * Castrate testosterone level (\<50ng/dl or 1.7nmol /L). * Radiological confirmation of metastatic disease OR -Progressive disease at study entry defined as one or more of the following criteria occurring in the setting of castrate levels of testosterone: --radiographic progression defined as any new or enlarging bone lesions or growing lymph node disease, consistent with prostate cancer OR * PSA progression defined by sequence of rising values separated by \>1 week (2 separate increasing values over a minimum of 1 ng/ml (Prostate Cancer Working Group 3 (PCWG3) PSA eligibility criteria). If participants had been on flutamide, PSA progression is documented 4 weeks or more after withdrawal. For participants on bicalutamide or nilutamide disease progression is documented 6 or more weeks after withdrawal. The requirement for a 4-6-week withdrawal period following discontinuation of flutamide, nilutamide, or bicalutamide only applies to participants who have been on these drugs for at least the prior 6 months. For all other participants, they must stop bicalutamide, nilutamide, or flutamide prior to treatment initiation. * Participants are willing to continue androgen deprivation therapy (ADT) with a gonadotropin-releasing hormone analog/antagonist or bilateral orchiectomy. * Age \>=18 years. * ECOG performance status \<= 1 * Participants must have adequate organ and marrow function as defined below: * Absolute neutrophil count \>= 1000/mcL * Platelets \>= 100,000/mcL * Hemoglobin \>= 9.0 g/dL * Total bilirubin within normal institutional limits; in participants with Gilbert s, \<= 3.0 mg/dL * AST (SGOT)/ALT (SGPT) \<= 2.5X upper limit of normal. For subjects with liver involvement in their tumor, AST \<= 3.5 (SqrRoot) ULN, ALT \<= 3.5 (SqrRoot) ULN, and bilirubin \<= 3.0 is acceptable * Creatinine within 1.5X upper limit of normal institutional limits * Men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) at the study entry, during the study, and maintain such contraception until 4 months following the last dose of any study agent. * Human immunodeficiency virus (HIV)-positive participants must have CD4 count \>= 200 cells per cubic millimeter before treatment initiation, be on stable antiretroviral therapy for at least 4 weeks and have no reported opportunistic infections or Castleman s disease within 12 months prior to treatment initiation. * Participants with successfully treated HCV if HCV viral load is undetectable. * Ability of subject to understand and the willingness to sign a written informed consent document EXCLUSION CRITERIA: * Participants who are immunocompromised as follows: * Chronic administration (defined as daily or every other day for continued use \>14 days) of systemic corticosteroids or other immune suppressive drugs, within 28 days before treatment initiation. NOTE: Systemic corticosteroids at physiologic doses \<= 10 mg/day of prednisone or equivalent and steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) are allowed. Nasal, or inhaled steroid, topical steroid creams, and eye drops for small body areas are also allowed. * History of allogeneic peripheral stem cell transplantation, or solid organ transplantation requiring immunosuppression. * Active autoimmune disease, except: participants with type 1 diabetes mellitus, vitiligo, psoriasis, hypo- or hyperthyroid disease not requiring current immunosuppression, or with other endocrine disorders on replacement hormones if the condition is well controlled. * History of prostate cancer with brain/leptomeningeal metastasis. Note: Except if status post definitive radiotherapy or surgery and are asymptomatic. * History of allergic reactions attributed to compounds of similar chemical or biologic composition to study agents to be used. * Known allergy to eggs, egg products, aminoglycoside antibiotics (e.g., gentamicin or tobramycin). * Prior investigational drug, chemotherapy, immunotherapy, or any prior radiotherapy (except for palliative bone directed therapy) within the past 28 days prior to treatment initiation. Note: except if all treatment-related toxicities have resolved or are minimal and the participant meets all other eligibility criteria. * Participants who have had cytotoxic chemotherapy for metastatic castration-resistant prostate cancer within the past year. NOTE: Participants who have had docetaxel for metastatic castration sensitive prostate cancer per CHAARTED data may enroll as long as they did not have progressive disease while on docetaxel and are 3 months removed from treatment, with all treatment-related toxicities resolving to at least grade 1. * Uncontrolled intercurrent acute or chronic illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure (\>New York Heart Association Class I), hepatic disease, unstable angina pectoris, serious cardiac arrhythmia, requiring medication, uncontrolled hypertension (SBP\>170/ DBP\>105) or psychiatric illness/social situations within 12 months before treatment initiation that would limit compliance with study requirements. * History of bleeding diathesis or recent major bleeding events (i.e. Grade \>= 2 bleeding events within 4 weeks prior to study treatment initiation) * Subjects unwilling to accept blood products or blood transfusions as medically indicated. As there is a risk of severe bleeding with M7824, participants must be willing to receive blood transfusions if medically necessary for their own safety * Subjects who received a live vaccine within 4 weeks or COVID-19 vaccines within 2 weeks prior to the study treatment initiation. * Use of herbal products that may decrease PSA levels (e.g., saw palmetto). * Major surgery within 4 weeks before treatment initiation. NOTE: A biopsy will not preclude a subject from starting the study. * History of hepatitis B (HBV) infection. * Use of regularly scheduled opiate analgesics for prostate cancer-related pain. * Subjects unwilling to accept blood products as medically indicated * Any condition which, in the opinion of the investigator, would prevent full participation in this trial (including the long-term follow-up), or would interfere with the evaluation of the trial endpoints.